Skip to main content
Skip table of contents

Disaster recovery documentation

1) VM Snapshots

Client should create VM snapshots from VM solution for all servers, periodically and save them to a different storage. This should be configured as per client backup policy.

Recommended is to do backup each day and keep the last 2 backups.

Restore can be done from the snapshots.

2) Data and configuration backup

Veridium has the following components, for which backups are done:

  • cassandra - retention default 2 backups, done each week.

  • elasticsearch - backup incremental.

  • zookeeper - backup weekly, 3 backups are kept

  • configuration data - main information files, in order to restore specific configuration (certificates, etc).

It is recommended that backup data should be mounted on a different disk, not a disk from the same machine. The default path where backups are kept is /opt/veridiumid/backup/. The path can be changed in configuration files.

Restore can be done from individual backups.

Data and configuration backup - details

Node Type

Backups performed

Persistence node

Cassandra backup

ElasticSearch backup

Configuration backup

Webapp nodes

Zookeeper backup

Configuration backup

1) Zookeeper backup

This process will download the content of Zookeeper and preserve it on the disk.

 

Details

Script location

/etc/veridiumid/scripts/zookeeper_backup.sh

Configuration file

Does not use a configuration file

Cronjob default time

0 2 * * 6 (Saturdays at 02:00)

Log file

/var/log/veridiumid/ops/zookeeper_backup.log

Number of backups maintained

5

Backup location

/opt/veridiumid/backup/zookeeper

Backup name

zookeeper_backup_YYYY-MM-DD-HH-mm-ss

Changing the number of maintained backups:

To change the limit you can run the following command:

CODE
sed -i "s|LIMIT=.*|LIMIT=<NEW_LIMIT>|g" /etc/veridiumid/scripts/zookeeper_backup.sh

Where:
<NEW_LIMIT> is the new number of backups that will be kept on the disk

Performing a backup:

Run the following command to perform a Zookeeper backup:

CODE
bash /etc/veridiumid/scripts/zookeeper_backup.sh

Recover from backup:

To recover the JSON files from a backup run the following command:

CODE
bash /opt/veridiumid/migration/bin/migration.sh -u PATH_TO_BACKUP

Where:
PATH_TO_BACKUP is the fullpath to the Zookeeper backup directory, for example: /opt/veriidumid/backup/zookeeper/zookeeper_backup_2023-06-17-02-00-01

2) Cassandra backup

This process will make a copy of existing Cassandra keyspaces and their configuration and preserve them on the disk.

 

Details

Script location

/opt/veridiumid/backup/cassandra/cassandra_backup.sh

Configuration file

/opt/veridiumid/backup/cassandra/cassandra_backup.conf

Cronjob default time

0 4 * * 6 (Saturdays at 04:00)

Log file

/var/log/veridiumid/cassandra/backup.log

Number of backups maintained

3

Backup location

/opt/veridiumid/backup/cassandra

Backup name

YYYY-MM-DD_HH-mm

Configuration file:

The configuration file contains the following:

CODE
CASSANDRA_HOME="/opt/veridiumid/cassandra"
CASSANDRA_KEYSPACE="veridium"
SNAPSHOT_RETENTION="3"
BACKUP_LOCATION="/opt/veridiumid/backup/cassandra"
LOG_DIRECTORY="/var/log/veridiumid/cassandra"
USER="veridiumid"
GROUP="veridiumid"

Changing the number of maintained backups:

To change the number of maintained backup change the value of SNAPSHOT_RETENTION in the configuration file.

Performing a backup:

Run the following command as root user to perform a Cassandra backup:

CODE
bash /opt/veridiumid/backup/cassandra/cassandra_backup.sh -c=/opt/veridiumid/backup/cassandra/cassandra_backup.conf

Recover from backup:

  • Stop webapplications on all webapp nodes:

    • CODE
      # Connect to all webapp nodes and run the following command as root:
      bash /etc/veridiumid/scripts/veridium_services.sh stop
  • Recreate the Cassandra keyspace

    • CODE
      # Run this command on a single persistence node as root:
      python3 ./cassandra-restore.py --debug --backup BACKUP --create
      
      # Where BACKUP is the full path to the backup directory, for example: /opt/veridiumid/backup/cassandra/2022-05-04_12-51
  • Restore the data from the backup

    • CODE
      # Run this command on all persistence nodes as root (can be run in parallel):
      python3 ./cassandra-restore.py --debug --backup BACKUP --restore
      
      # Where BACKUP is the full path to the backup directory, for example: /opt/veridiumid/backup/cassandra/2022-05-04_12-51
  • Rebuild indexes

    • CODE
      # Run this command on all persistence nodes as root (can be run in parallel):
      python3 ./cassandra-restore.py --debug --backup BACKUP --index
      
      # Where BACKUP is the full path to the backup directory, for example: /opt/veridiumid/backup/cassandra/2022-05-04_12-51
  • Start web applications on all webapp nodes

    • CODE
      # Connect to all webapp nodes and run the following command as root:
      bash /etc/veridiumid/scripts/veridium_services.sh start
  • Repair the Cassandra cluster

    • CODE
      # Run the following command as root on all persistence nodes, one at a time:
      bash /opt/veridiumid/cassandra/conf/cassandra_maintenance.sh -c /opt/veridiumid/cassandra/conf/maintenance.conf

3) ElasticSearch backup

This process will download the data existing in ElasticSearch and the settings for all indexes and save them on the disk.

 

Details

Script location

/opt/veridiumid/elasticsearch/bin/elasticsearch_backup.sh

Configuration file

/opt/veridiumid/elasticsearch/bin/elasticsearch_backup.conf

Cronjob default time

15 0 * * * (Everyday at 00:15)

Log file

/var/log/veridiumid/elasticsearch/backup.log

Number of backups maintained

10 (only for settings backups)

Backup location

/opt/veridiumid/backup/elasticsearch/data → for ElasticSearch data

/opt/veridiumid/backup/elasticsearch/settings → for ElasticSearch index settings

Backup name

YYYY-MM-DD_HH-mm → for ElasticSearch index settings

INDEX_NAME-YYYY-MM_DDHHmmss.esdb.tar.gz → for ElasticSearch Data

Configuration file:

The configuration file contains the following:

CODE
# Debug enabled or not (1 or 0)
DEBUG=0
# Path to directory used for Data backups
BKP_DATA_DIR=/opt/veridiumid/backup/elasticsearch/data
# Path to directory used for Settings backups
BKP_SETTINGS_DIR=/opt/veridiumid/backup/elasticsearch/settings
# Limit of settings backups
LIMIT=10
# Export page size
EXPORT_PAGE_SIZE=5000
# Request timeout in minutes
REQUEST_TIMEOUT=5
# Connection timeout in minutes
CONNECTION_TIMEOUT=3
# Number of parallel tasks
PARALLEL_TASKS=2

Changing the number of maintained settings backups:

To change the number of maintained settings backup change the value of LIMIT in the configuration file.

Performing a backup:

Run the following command as root user to perform an ElasticSearch backup:

CODE
bash /opt/veridiumid/elasticsearch/bin/elasticsearch_backup.sh /opt/veridiumid/elasticsearch/bin/elasticsearch_backup.conf

Recover from backup:

Run the following command as root to recover from an ElasticSearch backup:

CODE
bash /opt/veridiumid/migration/bin/elk_ops.sh --restore --dir=BACKUP_PATH
# Where: BACKUP_PATH is the full path to the backup directory

# Settings restore
bash /opt/veridiumid/migration/bin/elk_ops.sh --restore --dir=/opt/veridiumid/backup/elasticsearch/settings/YYYY-MM-DD_HH-mm

# All Data restore
bash /opt/veridiumid/migration/bin/elk_ops.sh --restore --dir=/opt/veridiumid/backup/elasticsearch/data

# Restore specific index data
bash /opt/veridiumid/migration/bin/elk_ops.sh --restore --dir=/opt/veridiumid/backup/elasticsearch/data/INDEX_NAME-YYYY-MM_DDHHmmss.esdb.tar.gz

4) Configuration backup

This process will make an archive containing all important configuration files for all VeridiumID services.

 

Details

Script location

/etc/veridiumid/scripts/backup_configs.sh

Configuration file

/etc/veridiumid/scripts/backup_configs.conf

Cronjob default time

0 2 * * 6 (Saturdays at 02:00)

Log file

/var/log/veridiumid/ops/backup_configs.log

Number of backups maintained

Last 31 days worth of backups

Backup location

/opt/veridiumid/backup/all_configs

Backup name

IP_ADDR_YYYYMMDDHHmmss.tar.gz

Configuration file:

The configuration file contains the list of files that will be backed up, in the following format: service_name:full_path:user:group

CODE
cassandra:/opt/veridiumid/cassandra/conf/cassandra-env.sh:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/cassandra_maintenance.sh:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/cassandra-rackdc.properties:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/cassandra.yaml:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/cqlsh_cassandra_cert.pem:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/cqlsh_cassandra_key.pem:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/jvm.options:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/jvm-server.options:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/KeyStore.jks:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/maintenance.conf:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/TrustStore.jks:ver_cassandra:veridiumid
cassandra:/opt/veridiumid/cassandra/conf/veridiumid_cqlshrc:ver_cassandra:veridiumid
elasticsearch:/opt/veridiumid/elasticsearch/config:veridiumid:veridiumid
fido:/opt/veridiumid/fido/conf/log4j-fido.xml:ver_fido:veridiumid
fido:/etc/default/veridiumid/ver_fido:ver_fido:veridiumid
haproxy:/opt/veridiumid/haproxy/conf/haproxy.cfg:ver_haproxy:veridiumid
haproxy:/opt/veridiumid/haproxy/conf/server.pem:ver_haproxy:veridiumid
haproxy:/opt/veridiumid/haproxy/conf/client-ca.pem:ver_haproxy:veridiumid
notifications:/etc/default/veridiumid/ver_notifications:ver_notifications:veridiumid
notifications:/opt/veridiumid/notifications/conf/log4j-notifications.xml:ver_notifications:veridiumid
selfservice:/etc/default/veridiumid/ver_selfservice:ver_selfservice:veridiumid
selfservice:/opt/veridiumid/selfservice/conf/log4j.xml:ver_selfservice:veridiumid
statistics:/etc/default/veridiumid/ver_statistics:ver_statistics:veridiumid
statistics:/opt/veridiumid/statistics/conf/log4j-auth-sessions-global.xml:ver_statistics:veridiumid
statistics:/opt/veridiumid/statistics/conf/log4j-auth-sessions-local.xml:ver_statistics:veridiumid
statistics:/opt/veridiumid/statistics/conf/log4j-dashboard-account-bioengines.xml:ver_statistics:veridiumid
statistics:/opt/veridiumid/statistics/conf/log4j-dashboard-sessions-global.xml:ver_statistics:veridiumid
statistics:/opt/veridiumid/statistics/conf/log4j-dashboard-account-sessions.xml:ver_statistics:veridiumid
statistics:/opt/veridiumid/statistics/conf/log4j-dashboard-sessions-local.xml:ver_statistics:veridiumid
tomcat:/opt/veridiumid/tomcat/bin/setenv.sh:ver_tomcat:veridiumid
tomcat:/opt/veridiumid/tomcat/certs:ver_tomcat:veridiumid
tomcat:/opt/veridiumid/tomcat/conf:ver_tomcat:veridiumid
zookeeper:/opt/veridiumid/zookeeper/conf/java.env:ver_zookeeper:veridiumid
zookeeper:/opt/veridiumid/zookeeper/conf/zoo.cfg:ver_zookeeper:veridiumid
zookeeper:/opt/veridiumid/zookeeper/data/myid:ver_zookeeper:veridiumid
data-retention:/etc/default/veridiumid/ver_data_retention:ver_data_retention:veridiumid
data-retention:/opt/veridiumid/data-retention/conf/log4j-data-retention.xml:ver_data_retention:veridiumid
freeradius:/opt/veridiumid/freeradius/etc/raddb/certs:ver_freeradius:veridiumid
freeradius:/opt/veridiumid/freeradius/etc/raddb/clients.conf:ver_freeradius:veridiumid
freeradius:/opt/veridiumid/freeradius/etc/raddb/sites-available/tls:ver_freeradius:veridiumid
kafka:/opt/veridiumid/kafka/config/certs:ver_kafka:veridiumid
kafka:/opt/veridiumid/kafka/config/server.properties:ver_kafka:veridiumid
opa:/opt/veridiumid/opa/certs:veridiumid:veridiumid
opa:/opt/veridiumid/opa/conf/opa.yaml:veridiumid:veridiumid
shibboleth-idp:/opt/veridiumid/shibboleth-idp/credentials:ver_tomcat:veridiumid
shibboleth-idp:/opt/veridiumid/shibboleth-idp/metadata:ver_tomcat:veridiumid
shibboleth-idp:/opt/veridiumid/shibboleth-idp/conf:ver_tomcat:veridiumid
shibboleth-idp:/opt/veridiumid/shibboleth-idp/edit-webapp:ver_tomcat:veridiumid
websecadmin:/etc/default/veridiumid/ver_websecadmin:ver_websecadmin:veridiumid
websecadmin:/opt/veridiumid/websecadmin/certs:ver_websecadmin:veridiumid
websecadmin:/opt/veridiumid/websecadmin/conf/log4j.xml:ver_websecadmin:veridiumid
migration:/opt/veridiumid/migration/conf/log4j.xml:veridiumid:veridiumid
system:/etc/hosts:root:root
system:/etc/sysctl.conf:root:root
system:/etc/security/limits.conf:root:root
ca:/opt/veridiumid/CA:veridiumid:veridiumid

Changing the number of maintained backups:

The value is hardcoded in the /etc/veridiumid/scripts/backup_configs.sh script. To modify it change the following line:

CODE
line 127: find ${VERIDIUM_DIR}/backup/all_configs -mtime +31 -exec rm -rf {} \;

And change 31 with the number of days you wish to keep backups.

Performing a backup:

Run the following command as root user to perform an Configuration backup:

CODE
bash /etc/veridiumid/scripts/backup_configs.sh /etc/veridiumid/scripts/backup_configs.conf

Recover from backup:

Run the following command as root to recover from an Configuration backup:

  • Stop all VeridiumID services

    • CODE
      # Connect to all nodes and use the following command as root, start with webapps and then persistence nodes:
      bash /etc/veridiumid/scripts/veridium_services.sh stop
  • Revert configuration files

    • CODE
      # Connect to all nodes and run the following command as root:
      bash /etc/veridiumid/scripts/config_revert.sh -c /etc/veridiumid/scripts/backup_configs.conf -b /opt/veridiumid/backup/all_configs/IP_ADDR_YYYYMMDDHHmmss.tar.gz
  • Start all VeridiumID services

    • CODE
      # Connect to all nodes and use the following command as root, start with persistence and then webapps nodes:
      bash /etc/veridiumid/scripts/veridium_services.sh start
JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.