v3.9.1 v3.9 v3.8.4 v3.8.3 v3.8.2 v3.8.1 v3.8 v3.7.2 v3.7.1 v3.7 v3.6
Auto Light Dark
Auto Light Dark
v3.9.1 v3.9 v3.8.4 v3.8.3 v3.8.2 v3.8.1 v3.8 v3.7.2 v3.7.1 v3.7 v3.6
Breadcrumbs

K8s/Openshift - Disaster Recovery Procedure

This document describes the procedure for restoring VeridiumID 3.9.1 from a backup. Follow all steps in the order listed to ensure a successful restoration.

Prerequisites

  • Install all components as described in the Install procedure

Step 1: Remove any potential Cassandra datacenter connection restriction

In the vid-maintenance pod, set the DATACENTER_TAG variable to an empty value, allowing the application to connect to any available Cassandra datacenter. The command changes only the DATACENTER_TAG line and keeps the other lines of the file.

sed -i 's/^DATACENTER_TAG=.*/DATACENTER_TAG=/' /etc/default/veridiumid/ver_migration

Step 2: Restore Zookeeper

Copy the Zookeeper backup archive to the maintenance pod:

kubectl -n $NAMESPACE cp backups/backup_zookeeper.tar.gz vid-maintenance-<POD_ID>:/tmp/backup_zookeeper.tar.gz

Run the Zookeeper restore script in the maintenance pod:

bash /scripts/restore-zookeeper.sh /tmp/backup_zookeeper.tar.gz

Step 3: Renew Cassandra Certificates

  1. In the vid-maintenance pod, execute the certificate renewal script:

    bash /scripts/renew-cassandra-certificates.sh

  2. Delete all Cassandra pods to allow them to restart with the renewed certificates.

Step 4: Renew Interservice Certificates

In the vid-maintenance pod, execute the interservice certificate renewal script:

bash /scripts/renew-interservice-certificates.sh

The script restarts vid-websecadmin, vid-opa, vid-websec, vid-adservice and vid-shibboleth.

Step 5: Update Keystore Passwords

Retrieve the old keystore password in the vid-maintenance pod:

jq .certStore.ca.pwd /tmp/zookeeper-restore/config.json | tr -d '"'

Update the following ConfigMaps with the retrieved password:

  • vid-websecadmin-config: WEBSECADMIN_KEYSTORE_PASS, WEBSECADMIN_TRUSTSTORE_PASS

  • vid-websec-tomcat-server-xml-config: in server-TOMCAT.xml

  • vid-adservice-tomcat-server-xml-config: in server-TOMCAT.xml

In server-TOMCAT.xml, update whichever attribute pair is present. Current Tomcat uses certificateKeystorePassword and truststorePassword; older builds use keystorePass and truststorePass.

Restart the affected deployments:

oc -n $NAMESPACE rollout restart deploy vid-adservice vid-websec vid-websecadmin vid-opa vid-shibboleth

Step 6: Recreate CA Secrets and Update HAProxy Certificates

Generate CA secrets (In the maintenance pod):

Bash
cd /tmp/zookeeper-restore/ && bash /scripts/create-ca-secrets.sh

Retrieve the ca.crt from the vid-ca-cert Secret and add it to vid-haproxy-certs as client-ca.pem.
Restart HAProxy. It runs as a DaemonSet or as a Deployment, depending on the environment. Restart the object that exists:

kubectl -n $NAMESPACE rollout restart daemonset/vid-haproxy
kubectl -n $NAMESPACE rollout restart deployment/vid-haproxy

Step 7: Restore Cassandra

Copy the Cassandra backup (cassandra-backup-<timestamp>.tar.gz, or .tar.gz.gpg if encrypted) to the first Cassandra StatefulSet pod. The pod name is <clusterName>-<datacenter>-default-sts-0, from k8ssandra-values.yaml:

kubectl -n $NAMESPACE cp backups/cassandra-backup-<timestamp>.tar.gz cassandra-<ENV_NO>-dc1-default-sts-0:/tmp/

If the archive is encrypted, also make the GPG private key and passphrase available in the pod at /tmp/private-key.asc and /tmp/passphrase (see the Backup Encryption section).

Inside the Cassandra pod, run the restore script. For disaster recovery of the whole cluster, pass --all to restore every tenant keyspace in the archive; pass a single keyspace name to restore just one tenant:

# full-cluster DR (all tenants):
bash /scripts/restore-cassandra.sh /tmp/cassandra-backup-<timestamp>.tar.gz --all
# single tenant:
bash /scripts/restore-cassandra.sh /tmp/cassandra-backup-<timestamp>.tar.gz <keyspace>

Repair the cluster (omit --ssl if the cluster runs without encryption):

nodetool --ssl repair --full

Step 8: Restore Elasticsearch and Kibana

Retrieve the Elasticsearch password from the maintenance pod:

Bash
echo $ELASTICSEARCH_PASSWORD

Update configuration files in Zookeeper (using WebsecAdmin - Advanced section):

  • elasticsearch.json

    • Set the password to the value retrieved above.

    • Set apiKey to an empty string.

  • kibana.json

    • Set apiKey to an empty string.

  • elasticsearch/index-templates.json

    • Set "number_of_replicas": 1.

Step 9: Reconfigure and Restore Elasticsearch

Copy the Elasticsearch backup archive to the maintenance pod:

kubectl -n $NAMESPACE cp backups/backup_elasticsearch.tar.gz vid-maintenance-<POD_ID>:/tmp/backup_elasticsearch.tar.gz

From the maintenance pod:

  1. Delete all Veridium indexes:

    Bash
    curl -k -X DELETE https://elastic:$ELASTICSEARCH_PASSWORD@$ELASTICSEARCH_HOST:9200/$(curl -k -s https://elastic:$ELASTICSEARCH_PASSWORD@$ELASTICSEARCH_HOST:9200/_cat/indices/veridium*?h=idx | tr '\n' ',')

  2. Recreate index definitions:

    /opt/veridiumid/migration/bin/elk_ops.sh --update-settings

  3. Restore Elasticsearch data from backup:

    bash /scripts/restore-elasticsearch.sh /tmp/backup_elasticsearch.tar.gz

  4. Migrate data from Cassandra to Elasticsearch:

    bash /scripts/migrate_to_elk.sh -a
bash /scripts/migrate_to_elk.sh -d
bash /scripts/migrate_to_elk.sh -p
bash /scripts/migrate_to_elk.sh -f

Last updated: