K8s/Openshift - Disaster Recovery Procedure

This document describes the procedure for restoring VeridiumID 3.8.4 from a backup. Follow all steps in the order listed to ensure a successful restoration.

Prerequisites

Backup configmaps and secrets:
1. download https://veridium-repo.veridium-dev.com/repository/helm-releases/veridiumid-containers/3.8.4/backup-configmaps-secrets.sh to the maintenance pod
2. execute it then download the backup to your workstation
Download https://veridium-repo.veridium-dev.com/repository/helm-releases/veridiumid-containers/3.8.4/restore-configmaps-secrets.sh to your workstation (not the maintenance pod)
(optional) Route configuration
Empty namespace and resources
Values files from git
(optional) Installation bundle: https://veridium-repo.veridium-dev.com/repository/helm-releases/veridiumid-containers/3.8.4/veridiumid-saas-3.8.4.zip
tar, unzip tools

Step 1: Restore configmaps and secrets - on the workstation

export NAMESPACE=<NAMESPACE>
bash restore-configmaps-secrets.sh <file>.tar.gz

If the vid-credentials-ecr secret does not exist after the restore, create it. The cluster uses this secret to authenticate with the Docker registry and pull the images.

Step 2: Install helm charts using values files from git

Create PVC

oc -n $NAMESPACE --validate=false apply -f values/pvc-backups.yaml || true

Launch operators

helm upgrade --install -n $NAMESPACE -f values/k8ssandra-operator-values.yaml --no-hooks k8ssandra-operator-$ENV_NO helm/k8ssandra-operator-1.20.2-r3.tgz
helm upgrade --install -n $NAMESPACE -f values/eck-operator-values.yaml eck-operator-$ENV_NO helm/eck-operator-2.1.0.tgz
helm upgrade --install -n $NAMESPACE -f values/zookeeper-operator-values.yaml zookeeper-operator-$ENV_NO helm/zookeeper-operator-0.2.15.tgz

Launch databases

helm upgrade --install -n $NAMESPACE -f values/k8ssandra-values.yaml --timeout 60m k8ssandra-$ENV_NO helm/vid-k8ssandra-0.9.0.tgz
helm upgrade --install -n $NAMESPACE -f values/elasticsearch-values.yaml elasticsearch-$ENV_NO helm/elasticsearch-0.3.1.tgz
helm upgrade --install -n $NAMESPACE --timeout 60m -f values/zookeeper-values.yaml zookeeper-$ENV_NO helm/zookeeper-0.2.15.tgz

Launch VeridiumID apps

helm upgrade --install -n $NAMESPACE -f values/veridiumid-values.yaml veridiumid helm/veridiumid-0.9.0.tgz

Step 3: Restore data from backups

Create helper variables (optional):

MAINTENANCE_POD=$(kubectl -n $NAMESPACE get pods -l app.kubernetes.io/name=vid-maintenance -o jsonpath='{.items[0].metadata.name}')

CASSANDRA_POD=cassandra-dev-zurich-default-sts-0

Copy backup files to pods

kubectl -n $NAMESPACE cp <file>.tar.gz $MAINTENANCE_POD:/tmp/backup_zookeeper.tar.gz
kubectl -n $NAMESPACE cp <file>.tar.gz $MAINTENANCE_POD:/tmp/backup_elasticsearch.tar.gz
kubectl -n $NAMESPACE cp <file>.tar.gz $CASSANDRA_POD:/tmp/backup_cassandra.tar.gz

Restore Zookeeper data

# create zookeeper path
oc -n $NAMESPACE exec zookeeper-dev-0 -- bin/zkCli.sh create /veridiumid ""
oc -n $NAMESPACE exec zookeeper-dev-0 -- bin/zkCli.sh create /veridiumid/12.4.72 ""

# restore data
oc -n $NAMESPACE exec deploy/vid-maintenance -- bash /scripts/restore-zookeeper.sh /tmp/backup_zookeeper.tar.gz

Restore Cassandra data

# Replace the datacenter name
oc -n $NAMESPACE exec -it $CASSANDRA_POD -c cassandra -- cqlsh -e "CREATE KEYSPACE IF NOT EXISTS veridium WITH replication = {'class': 'NetworkTopologyStrategy', '<DATACENTER_NAME>': '3'}  AND durable_writes = true;"

oc -n $NAMESPACE exec $CASSANDRA_POD -c cassandra -- bash /scripts/restore-cassandra.sh /tmp/backup_cassandra.tar.gz
oc -n $NAMESPACE exec $CASSANDRA_POD -c cassandra -- nodetool --ssl repair --full

Restore Elasticsearch data

oc -n "$NAMESPACE" exec -i "$MAINTENANCE_POD" -- bash -s <<'EOF'
set -euox pipefail
bash /opt/veridiumid/migration/bin/migration.sh -d /tmp/zk1 --decrypt
jq '.clientConnection.credentials.passwd = env.ELASTICSEARCH_PASSWORD | .clientConnection.apiKey = ""' /tmp/zk1/elasticsearch.json | sponge /tmp/zk1/elasticsearch.json
jq '.clientConnection.apiKey = ""' /tmp/zk1/kibana.json | sponge /tmp/zk1/kibana.json
jq '.templates[].content.template.settings.number_of_replicas = 1' /tmp/zk1/elasticsearch/index-templates.json | sponge /tmp/zk1/elasticsearch/index-templates.json
bash /opt/veridiumid/migration/bin/migration.sh -u /tmp/zk1
/opt/veridiumid/migration/bin/elk_ops.sh --update-settings
bash /scripts/restore-elasticsearch.sh /tmp/backup_elasticsearch.tar.gz
bash /scripts/migrate_to_elk.sh -a
bash /scripts/migrate_to_elk.sh -d
bash /scripts/migrate_to_elk.sh -p
bash /scripts/migrate_to_elk.sh -f
EOF

Last updated: September 30, 2026