Elasticsearch troubleshooting & useful commands

check status by running command:

CODE

bash /etc/veridiumid/scripts/check_services.sh

Status explained:

it presents the number of servers in the cluster (node.total = 6)
it presents the overall status: green, yellow, red
- green means that everything is fine
- yellow that not all nodes are up or some shards needs to be reallocated (the system is still functional)
- red - elasticsearch is not fully functional.
pending/unassing tasks
- the number of tasks (ex. moving shards from one server to other)

Procedure - If one datacenter is lost, elasticsearch might be unavailable in remaing datacenter. Below is the procedure, how to remove on datacenter from CDCR.

CODE

1. stop elasticsearch nodes:
systemctl stop ver_elasticsearch

2. Remove the state on all nodes:
rm -rf /opt/veridiumid/elasticsearch/data/_state 

3. set information about the cluster master nodes, accordingly
vi /opt/veridiumid/elasticsearch/config/elasticsearch.yml
cluster.initial_master_nodes: [ "dc1-node1", "dc1-node2", "dc1-node3" ]

4. start the cluster on these 3 nodes

5. create API key and update websecadmin -> elasticsearch.json:
eops -x=GET -p=/_security/api_key
eops -x=PUT -p=/_security/api_key -d='{"name":"veridium"}'

6. in second datacenter, when restored, please ensure that initial master nodes are commented and seeds are all 6.
also delete content:
systemctl stop ver_elasticsearch
rm -rf /opt/veridiumid/elasticsearch/data
and start services:
systemctl start ver_elasticsearch

Procedure - If one datacenter is lost, how to remove one datacenter completely from CDCR

CODE

1. stop elasticsearch nodes:
systemctl stop ver_elasticsearch

2. Remove the state on all nodes:
rm -rf /opt/veridiumid/elasticsearch/data/_state

3. set information about the cluster master nodes, accordingly
vi /opt/veridiumid/elasticsearch/config/elasticsearch.yml
cluster.initial_master_nodes: [ "dc1-node1", "dc1-node2", "dc1-node3" ]
also configure properly: discovery.seed_hosts with only these 3 nodes
comment the following lines:
##cluster.routing.allocation.awareness.force.zone.values: dc1, dc2

4. start the cluster on these 3 nodes

5. create API key and update websecadmin -> elasticsearch.json:
eops -x=GET -p=/_security/api_key
eops -x=PUT -p=/_security/api_key -d='{"name":"veridium"}'

6. set the number of replicas according to new infrastructure. 
eops -x=PUT -p=/veridium.*/_settings -d='{"index":{"number_of_replicas":1}}'

7. connect to cassandra and truncate info about the cluster:
cqlsh
truncate table veridium.cdcr_datacenters

Backup :

Backup is done incrementally, for the data and needs to be executed on one node, in crontab. For more details, see article Troubleshooting - Disaster Recovery

Useful commands:

CODE

## list the indices
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -l 

## get help for _cat commands
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat

## modify number of replicas
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=PUT -p=/veridium.*/_settings -d='{"index":{"number_of_replicas":3}}'

## get nodes, indices, allocation of shards
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/nodes?v
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/indices?v
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/allocation?v   ## here you can see how many shards need to be realocated
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/shards         ## see how shards are allocated to each node
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cluster/allocation/explain?pretty   ## shards realocation reasons

## delete an indices
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=DELETE -p=/veridiumid.sessions-2023-03

##make and index readable
eops -x=PUT -p=/veridium.sessions-2023-08/_settings -d='{"settings":{"index.blocks.write":true}}'
###clone an index
eops -x=POST -p=/veridium.sessions-2023-08/_clone/veridiumbck.sessions-2023-08
##make and index writtable
eops -x=PUT -p=/veridium.sessions-2023-08/_settings -d='{"settings":{"index.blocks.write":false}}'

## copy the content of an index to the new index.
eops -x=POST -p=/_reindex -d='{"source":{"index":"veridium.sessions_history-2023-06"},"dest":{"index":"veridiumbck.sessions_history-2023-06"}}'

eops -x=DELETE -p=/veridiumbck.sessions_history-2023-06
eops -x=POST -p=/veridium.*/_flush

## get api key
eops -x=GET -p=/_security/api_key

## create api key
eops -x=PUT -p=/_security/api_key -d='{"name":"veridium"}'

##show hidden indices:
eops -x=GET -p=/_cat/indices/.*?v

## get cluster master nodes
 eops -x=GET -p=/_cluster/state?filter_path=metadata.cluster_coordination.last_committed_config