Elasticsearch troubleshooting & useful commands

check status by running command:

CODE

bash /etc/veridiumid/scripts/check_services.sh

Status explained:

it presents the number of servers in the cluster (node.total = 6)
it presents the overall status: green, yellow, red
- green means that everything is fine
- yellow that not all nodes are up or some shards needs to be reallocated (the system is still functional)
- red - elasticsearch is not fully functional.
pending/unassing tasks
- the number of tasks (ex. moving shards from one server to other)

Procedure - If one datacenter is lost, elasticsearch might be unavailable in remaing datacenter. Below is the procedure, how to remove on datacenter from CDCR.

CODE

1. stop elasticsearch nodes:
systemctl stop ver_elasticsearch

2. Remove the state on all nodes:
rm -rf /opt/veridiumid/elasticsearch/data/_state 

3. set information about the cluster master nodes, accordingly
vi /opt/veridiumid/elasticsearch/config/elasticsearch.yml
cluster.initial_master_nodes: [ "dc1-node1", "dc1-node2", "dc1-node3" ]

4. start the cluster on these 3 nodes

5. create API key and update websecadmin -> elasticsearch.json:
eops -x=GET -p=/_security/api_key
eops -x=PUT -p=/_security/api_key -d='{"name":"veridium"}'

6. in second datacenter, when restored, please ensure that initial master nodes are commented and seeds are all 6.
also delete content:
systemctl stop ver_elasticsearch
rm -rf /opt/veridiumid/elasticsearch/data
and start services:
systemctl start ver_elasticsearch

Procedure - If one datacenter is lost, how to remove one datacenter completely from CDCR

CODE

1. stop elasticsearch nodes:
systemctl stop ver_elasticsearch

2. Remove the state on all nodes:
rm -rf /opt/veridiumid/elasticsearch/data/_state

3. set information about the cluster master nodes, accordingly
vi /opt/veridiumid/elasticsearch/config/elasticsearch.yml
cluster.initial_master_nodes: [ "dc1-node1", "dc1-node2", "dc1-node3" ]
also configure properly: discovery.seed_hosts with only these 3 nodes
comment the following lines:
##cluster.routing.allocation.awareness.force.zone.values: dc1, dc2

4. start the cluster on these 3 nodes

5. create API key and update websecadmin -> elasticsearch.json:
eops -x=GET -p=/_security/api_key
eops -x=PUT -p=/_security/api_key -d='{"name":"veridium"}'

6. set the number of replicas according to new infrastructure. 
eops -x=PUT -p=/veridium.*/_settings -d='{"index":{"number_of_replicas":1}}'

7. connect to cassandra and truncate info about the cluster:
cqlsh
truncate table veridium.cdcr_datacenters

Backup :

Backup is done incrementally, for the data and needs to be executed on one node, in crontab. For more details, see article Troubleshooting - Disaster Recovery

Useful commands:

CODE

## list the indices
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -l 

## Check Cluster Health
eops -x=GET -p=/_cluster/health?level=indices

## Inspect Shards
eops -x=GET -p="/_cat/shards?v=true&h=index,shard,prirep,state,docs,store,node,unassigned.reason&s=state:desc"

##Identify unassigned shards
eops -x=GET -p="/_tasks?detailed=true&actions=*shard,*management"

##explain allocation
eops -x=GET -p="/_cluster/allocation/explain?pretty" -d='{"index":"dev682511251310.fido_devices-000001","shard":0,"primary":false}'

## get help for _cat commands
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat

## modify number of replicas
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=PUT -p=/veridium.*/_settings -d='{"index":{"number_of_replicas":3}}'

## get nodes, indices, allocation of shards
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/nodes?v
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/indices?v
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/allocation?v   ## here you can see how many shards need to be realocated
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cat/shards         ## see how shards are allocated to each node
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=GET -p=/_cluster/allocation/explain?pretty   ## shards realocation reasons

## delete an indices
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh -x=DELETE -p=/veridiumid.sessions-2023-03

##make and index readable
eops -x=PUT -p=/veridium.sessions-2023-08/_settings -d='{"settings":{"index.blocks.write":true}}'
###clone an index
eops -x=POST -p=/veridium.sessions-2023-08/_clone/veridiumbck.sessions-2023-08
##make and index writtable
eops -x=PUT -p=/veridium.sessions-2023-08/_settings -d='{"settings":{"index.blocks.write":false}}'

## copy the content of an index to the new index.
eops -x=POST -p=/_reindex -d='{"source":{"index":"veridium.sessions_history-2023-06"},"dest":{"index":"veridiumbck.sessions_history-2023-06"}}'

eops -x=DELETE -p=/veridiumbck.sessions_history-2023-06
eops -x=POST -p=/veridium.*/_flush

## get api key
eops -x=GET -p=/_security/api_key

## create api key
eops -x=PUT -p=/_security/api_key -d='{"name":"veridium"}'

##show hidden indices:
eops -x=GET -p=/_cat/indices/.*?v

## get cluster master nodes
 eops -x=GET -p=/_cluster/state?filter_path=metadata.cluster_coordination.last_committed_config

CODE

## REAL CASE EXAMPLE OF REINDEXING
## modify zookeeper content to be able to process elasticsearch data
eops -l | grep sessions-2025-12

eops -x=PUT -p=/veridium.sessions-2025-12/_settings -d='{"settings":{"index.blocks.write":true}}'
eops -x=POST -p=/veridium.sessions-2025-12/_clone/veridium.sessions-2025-12-02
## cloning operation took less then 10 secondes.
## with copy index, it did not work, as it was not able to convert the data
eops -l | grep sessions-2025-12
eops -x=POST -p=/_aliases -d='{"actions":[{"add":{"index":"veridium.sessions-2025-12-02","alias":"veridium.sessions"}}]}'

eops -x=DELETE -p=/veridium.sessions-2025-12

eops -x=PUT -p=/veridium.sessions-2025-12-02/_settings -d='{"index":{"number_of_replicas":3}}'