K8s/Openshift - Backup and Restore Guide

1. Backup

1.1. Automatic Backup

Automatic backups are configured to run at specified intervals defined in the values files. Each backup generates an archive which is then uploaded to a remote server using SCP. For Cassandra the archive is named cassandra-backup-<timestamp>.tar.gz (or cassandra-backup-<timestamp>.tar.gz.gpg when encryption is enabled — see 1.3. Backup Encryption). The backup schedule and remote server connection settings are specified in the following values files:

The schedule settings are:

The SCP settings are:

The shipped veridiumid-values.yaml has no upload blocks. Add one under each service that you enable, for example:

vid-maintenance:
  backup:
    zookeeper:
      enable: true
      schedule: "0 23 * * *"
      upload:
        user: backup
        address: 10.0.0.2
        path: /backup/zookeeper/

The SSH key for the remote server is in two Kubernetes secrets, under the key backup.key:

• k8ssandra-<ENV_NO>-backup-key, for Cassandra. When you set key, the chart creates this secret. When key is empty, you create it.

• veridiumid-backup-key, for Zookeeper, Elasticsearch and Configmaps/Secrets. You always create this secret.

1.1.1. Incremental Elasticsearch Backup

Elasticsearch backups are incremental by default. The first backup contains all the documents and is named <timestamp>_FULL.tar.gz. Each backup after that contains only the new documents and is named <timestamp>_INCR-from-<date>.tar.gz.

The job keeps the date of the last successful backup in a ConfigMap in the tenant namespace:

vid-maintenance:
  backup:
    elasticsearch:
      incremental:
        enable: true
        stateConfigMap: vid-maintenance-es-backup-state

To make the next backup contain all the documents again, delete the ConfigMap. To make every backup contain all the documents, set enable: false.

kubectl -n <namespace> get configmap vid-maintenance-es-backup-state -o yaml
kubectl -n <namespace> delete configmap vid-maintenance-es-backup-state

Limitations:

• An incremental archive does not contain the documents that changed without a new @timestamp, and it does not contain a record of the deleted documents.

• To restore, you must use the full archive and all the incremental archives that come after it. Keep them all on the remote server.

1.2. On-Demand Backup (optional)

1.2.1. Cassandra

To perform an on-demand backup for Cassandra, execute the following command inside a Cassandra pod. The pod name is <clusterName>-<datacenter>-default-sts-<N>, from k8ssandra-values.yaml. For example: cassandra-<ENV_NO>-dc1-default-sts-0:

/scripts/backup-cassandra.sh

1.2.2. Zookeeper

To perform an on-demand backup for Zookeeper, execute the following command inside the vid-maintenance pod:

/scripts/zookeeper_backup.sh

1.2.3. Elasticsearch

To perform an on-demand backup for Elasticsearch, execute the following command inside the vid-maintenance pod:

/scripts/elasticsearch_backup.sh

1.2.4. Configmaps and Secrets

To perform an on-demand backup for Configmaps and Secrets, execute the following command inside the vid-maintenance pod:

/scripts/backup-configmaps-secrets.sh

1.3. Backup Encryption (optional)

GPG can encrypt each backup archive with the public key of a recipient. The encrypted archive has the suffix .tar.gz.gpg. Configure the encryption in two places:

Both locations take the same settings:

The secret named by keysSecret must contain:

To restore an encrypted archive, the restore script needs the private key and the passphrase at /tmp/private-key.asc and /tmp/passphrase. The vid-maintenance pod mounts them from keysSecret. In the Cassandra pod, mount the keysSecret, or copy the two files in by hand. The restore script finds from the chart values whether the archive is encrypted.

2. Restore

2.1. Cassandra

1. Download the backup archive (cassandra-backup-<timestamp>.tar.gz, or .tar.gz.gpg if encrypted) from the remote server to the /tmp folder of a Cassandra pod (<clusterName>-<datacenter>-default-sts-<N>, for example cassandra-<ENV_NO>-dc1-default-sts-0).

2. If the archive is encrypted, make the GPG private key and passphrase available at /tmp/private-key.asc and /tmp/passphrase (see 1.3. Backup Encryption).

3. Execute the following command to restore Cassandra, passing the archive path and the target keyspace:

# Second argument selects what to restore from the (multi-tenant) archive:
#   <keyspace>  restore a single tenant keyspace
#   --all       restore every tenant keyspace in the archive (full-cluster DR)
/scripts/restore-cassandra.sh /tmp/cassandra-backup-<timestamp>.tar.gz <keyspace>
nodetool --ssl repair --full   # omit --ssl if the cluster runs without encryption

The backup archive contains every tenant keyspace. A tenant's keyspace name equals its tenant id; list what a backup contains with cqlsh --execute="desc keyspaces" on the cluster. Pass a single keyspace name to restore one tenant, or --all to restore the whole cluster (each keyspace restored in turn; a failure on one keyspace is reported and the rest still run).

2.2. Zookeeper

1. Download the backup archive from the remote server to the vid-maintenance pod (typically place it in the /tmp folder).

2. Execute the following command to restore Zookeeper, specifying the path to the backup archive:

/scripts/restore-zookeeper.sh <path/to/backup.tar.gz>

2.3. Elasticsearch

1. Create index templates

   /opt/veridiumid/migration/bin/elk_ops.sh --update-settings
   
   

2. Download the backup archives from the remote server to the vid-maintenance pod (typically place them in the /tmp folder). Backups are incremental by default (see 1.1.1. Incremental Elasticsearch Backup), so download the full archive and all the incremental archives that come after it.

3. Execute the following command to restore Elasticsearch, giving the archives oldest first:

   /scripts/restore-elasticsearch.sh /tmp/<date>_FULL.tar.gz /tmp/<date>_INCR-from-<date>.tar.gz
   
   

4. Copy any remaining data from Cassandra to Elasticsearch

   bash /scripts/migrate_to_elk.sh -a
   bash /scripts/migrate_to_elk.sh -d
   bash /scripts/migrate_to_elk.sh -p
   bash /scripts/migrate_to_elk.sh -f
   
   

2.4. Configmaps and Secrets

1. Download the backup archive from the remote server to the vid-maintenance pod (typically place it in the /tmp folder).

2. Execute the following commands to restore Configmaps and Secrets, specifying the path to the backup archive:

/scripts/restore-configmaps-secrets.sh <path/to/backup.tar.gz>

Last updated: October 15, 2026