Elasticsearch troubleshooting & useful commands
check status by running command:
bash /etc/veridiumid/scripts/check_services.sh
![](../../__attachments/3554706631/image-20230725-122035.png?inst-v=ffa2d127-74f7-4a38-ac00-bc62b4eb8eb4)
Status explained:
it presents the number of servers in the cluster (node.total = 6)
it presents the overall status: green, yellow, red
green means that everything is fine
yellow that not all nodes are up or some shards needs to be reallocated (the system is still functional)
red - elasticsearch is not fully functional.
pending/unassing tasks
the number of tasks (ex. moving shards from one server to other)
Procedure - If one datacenter is lost, elasticsearch might be unavailable in remaing datacenter. Below is the procedure, how to remove on datacenter from CDCR.
1. stop elasticsearch nodes:
systemctl stop ver_elasticsearch
2. Remove the state on all nodes:
rm -rf /opt/veridiumid/elasticsearch/data/_state
3. set information about the cluster master nodes, accordingly
vi /opt/veridiumid/elasticsearch/config/elasticsearch.yml
cluster.initial_master_nodes: [ "dc1-node1", "dc1-node2", "dc1-node3" ]
4. start the cluster on these 3 nodes
5. create API key and update websecadmin -> elasticsearch.json:
eops -x=GET -p=/_security/api_key
eops -x=PUT -p=/_security/api_key -d='{"name":"veridium"}'
6. in second datacenter, when restored, please ensure that initial master nodes are commented and seeds are all 6.
also delete content:
systemctl stop ver_elasticsearch
rm -rf /opt/veridiumid/elasticsearch/data
and start services:
systemctl start ver_elasticsearch
Procedure - If one datacenter is lost, how to remove one datacenter completely from CDCR
1. stop elasticsearch nodes:
systemctl stop ver_elasticsearch
2. Remove the state on all nodes:
rm -rf /opt/veridiumid/elasticsearch/data/_state
3. set information about the cluster master nodes, accordingly
vi /opt/veridiumid/elasticsearch/config/elasticsearch.yml
cluster.initial_master_nodes: [ "dc1-node1", "dc1-node2", "dc1-node3" ]
also configure properly: discovery.seed_hosts with only these 3 nodes
comment the following lines:
##cluster.routing.allocation.awareness.force.zone.values: dc1, dc2
4. start the cluster on these 3 nodes
5. create API key and update websecadmin -> elasticsearch.json:
eops -x=GET -p=/_security/api_key
eops -x=PUT -p=/_security/api_key -d='{"name":"veridium"}'
6. set the number of replicas according to new infrastructure.
eops -x=PUT -p=/veridium.*/_settings -d='{"index":{"number_of_replicas":1}}'
7. connect to cassandra and truncate info about the cluster:
cqlsh
truncate table veridium.cdcr_datacenters
Backup :
Backup is done incrementally, for the data and needs to be executed on one node, in crontab. For more details, see article Troubleshooting - Disaster Recovery
Useful commands:
## list the indices
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -l
## get help for _cat commands
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat
## modify number of replicas
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=PUT -p=/veridium.*/_settings -d='{"index":{"number_of_replicas":3}}'
## get nodes, indices, allocation of shards
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/nodes?v
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/indices?v
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/allocation?v ## here you can see how many shards need to be realocated
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/shards ## see how shards are allocated to each node
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cluster/allocation/explain?pretty ## shards realocation reasons
## delete an indices
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=DELETE -p=/veridiumid.sessions-2023-03
##make and index readable
eops -x=PUT -p=/veridium.sessions-2023-08/_settings -d='{"settings":{"index.blocks.write":true}}'
###clone an index
eops -x=POST -p=/veridium.sessions-2023-08/_clone/veridiumbck.sessions-2023-08
##make and index writtable
eops -x=PUT -p=/veridium.sessions-2023-08/_settings -d='{"settings":{"index.blocks.write":false}}'
## copy the content of an index to the new index.
eops -x=POST -p=/_reindex -d='{"source":{"index":"veridium.sessions_history-2023-06"},"dest":{"index":"veridiumbck.sessions_history-2023-06"}}'
eops -x=DELETE -p=/veridiumbck.sessions_history-2023-06
eops -x=POST -p=/veridium.*/_flush
## get api key
eops -x=GET -p=/_security/api_key
## create api key
eops -x=PUT -p=/_security/api_key -d='{"name":"veridium"}'
##show hidden indices:
eops -x=GET -p=/_cat/indices/.*?v
## get cluster master nodes
eops -x=GET -p=/_cluster/state?filter_path=metadata.cluster_coordination.last_committed_config