Cassandra troubleshooting & useful commands

check status, by running command:

CODE

bash /etc/veridiumid/scripts/check_services.sh

the status of each process are self explanatory - “is starting”, “is stopped”, “not enabled”, “is started”, “is not present on this node”
Cassandra in this example is a 3 node cluster.
- first node has status DN - Down Normal
- the next 2 nodes are UN - UP Normal
Normally, all 3 nodes should have status UN.
If the script throws errors, it means that it can not connect to Cassandra on this specific server.

2. Disk: check if there is available space on the disk. It might be that Cassandra it remained out of space during some maintenance operations. The disk occupation for Cassandra should be less than 75% at any time.

To be more precise, it should be free on each node the double size of the database (one time it needs for operation maintenance and one time for backup). The size of Cassandra is shown in the prinscreen as Load (5.69gb).

CODE

df -h

3. Memory:

There should be at least 1Gb of memory free on cassandra servers. To check this, run below command. In the exampke there are 11404Mb free.

CODE

free -m

4. Backup and maintenance

Backup and maintenance are running once per week per each server. They are running under root account, using the following scripts:

CODE

## cassandra repair cluster for inconsistencies
bash /opt/veridiumid/cassandra/conf/cassandra_maintenance.sh -c /opt/veridiumid/cassandra/conf/maintenance.conf
## cassandra compact each node, to reduse space and the number of tumbstones
bash /opt/veridiumid/cassandra/conf/cassandra_maintenance.sh -c /opt/veridiumid/cassandra/conf/maintenance.conf -k
## cassandra backup; in file /opt/veridiumid/backup/cassandra/cassandra_backup.conf there are kept the number of copies.
bash /opt/veridiumid/backup/cassandra/cassandra_backup.sh -c=/opt/veridiumid/backup/cassandra/cassandra_backup.conf

Backups are kept in this location:

CODE

/opt/veridiumid/backup/cassandra

The following logs should be monitored for Errors, to be sure that maintenance and backup jobs are finished successfully.

CODE

/var/log/veridiumid/cassandra/backup.log
/var/log/veridiumid/cassandra/maintenance.log

5. How to import a table from one environment to a different one:

CODE

1. take the cassandra backup from production and copy it to lab environment.

cassandra backups can be found here: /opt/veridiumid/backup/cassandra/
The tables that might be sincronzed should be:
- device_by_cert_uuid 
- device 
- identity 
- account 
- session_finished 
- history


copy the following files from production backup to lab environemnt:
please fill in the correct date of the backup
/opt/veridiumid/backup/cassandra/2023-05-31_13-26/*/veridium*/device_by_cert_uuid-*/snapshots/*/*db
/opt/veridiumid/backup/cassandra/2023-05-31_13-26/*/veridium*/device-*/snapshots/*/*db
/opt/veridiumid/backup/cassandra/2023-05-31_13-26/*/veridium*/identity-*/snapshots/*/*db
/opt/veridiumid/backup/cassandra/2023-05-31_13-26/*/veridium*/account-*/snapshots/*/*db
/opt/veridiumid/backup/cassandra/2023-05-31_13-26/*/veridium*/session_finished-*/snapshots/*/*db
/opt/veridiumid/backup/cassandra/2023-05-31_13-26/*/veridium*/history-*/snapshots/*/*db

2. ensure that there is sufficient space on lab environemnt to load data.
The files should be copy to these locations:
/opt/veridiumid/cassandra/data/veridium/device_by_cert_uuid*/
/opt/veridiumid/cassandra/data/veridium/device*/
/opt/veridiumid/cassandra/data/veridium/identity*/
/opt/veridiumid/cassandra/data/veridium/account*/
/opt/veridiumid/cassandra/data/veridium/session_finished*/
/opt/veridiumid/cassandra/data/veridium/history*/

3. assign proper permissions:
chown -R ver_cassandra.veridiumid /opt/veridiumid/cassandra/data/veridium/



4. load new data:
#first increase memory parameters for loader
sed 's|256M|1024M|g' /opt/veridiumid/cassandra/bin/sstableloader
## and after load the data
/opt/veridiumid/cassandra/bin/sstableloader -f /opt/veridiumid/cassandra/conf/cassandra.yaml -d `hostname -I` -p 9042 /opt/veridiumid/cassandra/data/veridium/device_by_cert_uuid-*/
/opt/veridiumid/cassandra/bin/sstableloader -f /opt/veridiumid/cassandra/conf/cassandra.yaml -d `hostname -I` -p 9042 /opt/veridiumid/cassandra/data/veridium/device-*/
/opt/veridiumid/cassandra/bin/sstableloader -f /opt/veridiumid/cassandra/conf/cassandra.yaml -d `hostname -I` -p 9042 /opt/veridiumid/cassandra/data/veridium/identity-*/
/opt/veridiumid/cassandra/bin/sstableloader -f /opt/veridiumid/cassandra/conf/cassandra.yaml -d `hostname -I` -p 9042 /opt/veridiumid/cassandra/data/veridium/account-*/
/opt/veridiumid/cassandra/bin/sstableloader -f /opt/veridiumid/cassandra/conf/cassandra.yaml -d `hostname -I` -p 9042 /opt/veridiumid/cassandra/data/veridium/session_finished-*/
/opt/veridiumid/cassandra/bin/sstableloader -f /opt/veridiumid/cassandra/conf/cassandra.yaml -d `hostname -I` -p 9042 /opt/veridiumid/cassandra/data/veridium/history-*/

6. Other useful commands:

CODE

## check the actual size of cassandra database on the disk
du -sh /opt/veridiumid/cassandra/data/
## check the actual size of each backup
du -sh /opt/veridiumid/backup/cassandra/20*
## in case of emergency, it can be removed one old backup, as in below example
#rm -rf /opt/veridiumid/backup/cassandra/2023-05-27_13-26

All below operations can be done online, without downtime, in case of an issue.

CODE

########## you can monitor what compaction are running on DB
/opt/veridiumid/cassandra/bin/nodetool compactionstats

############# deletes deleted data; maybe usefull when you want to clean data that was deleted recently.
/opt/veridiumid/cassandra/bin/nodetool garbagecollect veridium

############# rebuild index
/opt/veridiumid/cassandra/bin/nodetool rebuild_index veridium session_finished session_finished_index

############# operations on tables
/opt/veridiumid/cassandra/bin/nodetool repair -local veridium session_finished
/opt/veridiumid/cassandra/bin/nodetool compact veridium session_finished

######### show nodes in a cluster
/opt/veridiumid/cassandra/bin/nodetool describecluster

########  clear snashots
/opt/veridiumid/cassandra/bin/nodetool clearsnapshot veridium

######### get memory utilisation
/opt/veridiumid/cassandra/bin/nodetool info

In order to see a table description, to run a query in DB, the following command can be used:

CODE

/opt/veridiumid/cassandra/bin/cqlsh --cqlshrc=/opt/veridiumid/cassandra/conf/veridiumid_cqlshrc --ssl

use veridium;
expand on;
desc tables;
select * from identity limit 10;