II. Installation

1. Prerequisites

1.1. Client machine

Please install the following software on the machine that will be used to deploy VeridiumID:

Helm 3. Please check your installed version by running helm version
OpenShift CLI (oc) 4.11

1.2. Cluster

a) To ensure sufficient capacity and performance, the recommended minimum configuration for the cluster is to have three worker nodes with a combined total of at least 24 CPUs and 48GB of RAM. Each worker node needs to provide a minimum of 100GB local storage, in addition to an external volume capacity (EBS or similar) of 500GB that is necessary for data persistence. Please note that the resources allocated to the master nodes should be separate from the ones mentioned above.

b) About Operators: In order to deploy zookeeper, elasticsearch, cassandra in the namespace an operator will be installed for each one and the watchNamespace will be only the namespace created above. Please be sure you don't have an external operator to watch the namespace where the application will be installed.

c) Please create a StorageClass with "reclaimPolicy: Retain" + ReadWriteOnce(RWO) access mode + Encrypted (optional), this will be used for Cassandra Deployment / ElasticSearch Deployment / Zookeeper Deployment. Choose a name for this StorageClass. Marked above with "<STORAGECLASS_PERSISTENCE>"

d) Please create another StorageClass with "reclaimPolicy: Retain" + ReadWriteMany(RWX) access mode and + Encrypted (optional), this will be used for Zookeeper, ElasticSearch Backup.

2. Download configuration files

Please download veridiumid-saas-3.6.0.zip then unpack it by executing:

CODE

wget --user <NEXUS_USER> --password <NEXUS_PWD> https://veridium-repo.veridium-dev.com/repository/helm-releases/veridiumid-containers/3.6.0/veridiumid-saas-3.6.0.zip
unzip -o veridiumid-saas-3.6.0.zip -d 3.6.0
cd 3.6.0

Next customize the namespace and Cassandra datacenter name in configuration files:

CODE

sed -i 's|<NAMESPACE>|YOUR_NAMESPACE|g' values/*.yaml
# example: sed -i 's|<NAMESPACE>|dev1|g' values/*.yaml
sed -i 's|<DATACENTER_NAME>|YOUR_DATACENTER_NAME|g' values/*.yaml
# example: sed -i 's|<DATACENTER_NAME>|dc1|g' values/*.yaml
sed -i 's|<ENV_NO>|YOUR_ENV_NUMBER|g' values/*.yaml
# example: sed -i 's|<ENV_NO>|1|g' values/*.yaml
sed -i 's|<STORAGECLASS_PERSISTENCE>|YOUR_STORAGE_CLASS_ENCRYPTED|g' values/*.yaml
# example: sed -i 's|<STORAGECLASS_PERSISTENCE>|storage-encrypted|g' values/*.yaml
sed -i 's|<STORAGECLASS_BACKUPS>|STORAGE_CLASS_USED_FOR_BACKUPS|g' values/*.yaml
# example: sed -i 's|<STORAGECLASS_BACKUPS>|efs-sc|g' values/*.yaml

3. Before installation

3.1. Create a Secret with name "vid-credentials-ecr" with Username/Password in order to have access to your Private Docker Registry to pull images

In "default" namespace you need to create a "Secret" with exact name "vid-credentials-ecr".

CODE

oc create secret docker-registry vid-credentials-ecr --docker-server=<DOCKER_REGISTRY_SERVER> --docker-username=<DOCKER_USER> --docker-password=<DOCKER_PASSWORD> --docker-email=<DOCKER_EMAIL>
# example: oc create secret docker-registry vid-credentials-ecr --docker-server=registryhub.myregistry.com --docker-username=dockeruser --docker-password=dockerpass --docker-email=docker@myenv.com -n mynamespace

Replace our ECR Registry (018397616607.dkr.ecr.eu-central-1.amazonaws.com) with your registry in all files from veridium-containters folder.

CODE

sed -i 's|018397616607.dkr.ecr.eu-central-1.amazonaws.com|YOUR_REGISTRY_URL|g' values/*.yaml

3.2. Install Custom Resource Definitions for ECK operator

CODE

oc create -f values/veridiumid-crds/eck-operator-crds.yaml
# validate:
oc get crd | grep elastic
#output will be:
agents.agent.k8s.elastic.co                            XXXX-XXXX-XXTXX:XX:XXZ
apmservers.apm.k8s.elastic.co                          XXXX-XXXX-XXTXX:XX:XXZ
beats.beat.k8s.elastic.co                              XXXX-XXXX-XXTXX:XX:XXZ
elasticmapsservers.maps.k8s.elastic.co                 XXXX-XXXX-XXTXX:XX:XXZ
elasticsearchautoscalers.autoscaling.k8s.elastic.co    XXXX-XXXX-XXTXX:XX:XXZ
elasticsearches.elasticsearch.k8s.elastic.co           XXXX-XXXX-XXTXX:XX:XXZ
enterprisesearches.enterprisesearch.k8s.elastic.co     XXXX-XXXX-XXTXX:XX:XXZ
kibanas.kibana.k8s.elastic.co                          XXXX-XXXX-XXTXX:XX:XXZ
stackconfigpolicies.stackconfigpolicy.k8s.elastic.co   XXXX-XXXX-XXTXX:XX:XXZ

3.3. Install Custom Resource Definitions for ECK Zookeeper

CODE

oc create -f values/veridiumid-crds/zookeeper.yaml
# validate:
oc get crd | grep zookeeper
#output will be:
zookeeperclusters.zookeeper.pravega.io                 XXXX-XXXX-XXTXX:XX:XXZ

3.4. Install Custom Resource Definitions for K8ssandra Operator

CODE

oc create -f values/veridiumid-crds/k8ssandra-operator.yaml
oc get crd | grep k8ssandra
#output will be:
cassandrabackups.medusa.k8ssandra.io                   XXXX-XXXX-XXTXX:XX:XXZ
cassandrarestores.medusa.k8ssandra.io                  XXXX-XXXX-XXTXX:XX:XXZ
cassandratasks.control.k8ssandra.io                    XXXX-XXXX-XXTXX:XX:XXZ
clientconfigs.config.k8ssandra.io                      XXXX-XXXX-XXTXX:XX:XXZ
k8ssandraclusters.k8ssandra.io                         XXXX-XXXX-XXTXX:XX:XXZ
medusabackupjobs.medusa.k8ssandra.io                   XXXX-XXXX-XXTXX:XX:XXZ
medusabackups.medusa.k8ssandra.io                      XXXX-XXXX-XXTXX:XX:XXZ
medusabackupschedules.medusa.k8ssandra.io              XXXX-XXXX-XXTXX:XX:XXZ
medusarestorejobs.medusa.k8ssandra.io                  XXXX-XXXX-XXTXX:XX:XXZ
medusatasks.medusa.k8ssandra.io                        XXXX-XXXX-XXTXX:XX:XXZ
reapers.reaper.k8ssandra.io                            XXXX-XXXX-XXTXX:XX:XXZ
replicatedsecrets.replication.k8ssandra.io             XXXX-XXXX-XXTXX:XX:XXZ
stargates.stargate.k8ssandra.io                        XXXX-XXXX-XXTXX:XX:XXZ

3.5. Install CertManager - this is a requirement of k8ssandra-operator

If you already have CertManager installed in your Cluster, you can skip this step.

CODE

helm upgrade --install -f values/cert-manager-values.yaml cert-manager helm/cert-manager-v1.12.3.tgz --set installCRDs=true

4. Namespace where you want to install VeridiumID

4.1. Create a namespace

CODE

# create the namespace
oc apply -f values/namespace.yaml

4.2. Create a Secret with name "vid-credentials-ecr" with Username/Password in order to have access to your Private Docker Registry to pull images

CODE

oc create secret docker-registry vid-credentials-ecr --docker-server=<DOCKER_REGISTRY_SERVER> --docker-username=<DOCKER_USER> --docker-password=<DOCKER_PASSWORD> --docker-email=<DOCKER_EMAIL> -n <NAMESPACE>

4.3. Create a PersistentVolumeClaim that will be used for backing up Zookeeper/ElasticSearch

CODE

oc -n <NAMESPACE> apply -f values/pvc-backups.yaml

5. Deploy persistence charts

5.1. Cassandra

Edit the values/k8ssandra-values.yaml file and set the following values:

Property	Description	Recommended value
reaper.autoScheduling.enabled	Enable or disable the reaper to scheduling the repair/compact	Default: true
medusa.enable	If true, Medusa container will be launched.	false
medusa.storageProperties.podStorage.storageClassName	The type of storage to be provisioned for storing Medusa data	<STORAGECLASS_PERSISTENCE>
medusa.storageProperties.podStorage.size	The size of the disks where Medusa data will be stored	100Gi
medusa.storageProperties.maxBackupCount	Maximum number of backups to keep (used by the purge process).	2
medusa.storageProperties.maxBackupAge	Maximum backup age that the purge process should observe.	0
medusa_backup.enable	Enable or disable backup Cassandra using medusa	Default: true
medusa_backup.MedusaBackupSchedule.cronSchedule	Schedule when Medusa Backup will run	Default: 0 0 * * *
medusa_backup.MedusaBackupSchedulePurge.schedule	Schedule when purge job will run	Default: 0 1 * * *
cassandra.clusterName	Cassandra ClusterName	cassandra-<ENV_NO>
cassandra.datacenters[].name	Name of the datacenter	dc1
cassandra.datacenters[].size	Number of Cassandra nodes in cluster	3
cassandra.storageConfig.cassandraDataVolumeClaimSpec.storageClassName	The type of storage to be provisioned for storing Cassandra data	<STORAGECLASS_PERSISTENCE>
cassandra.storageConfig.cassandraDataVolumeClaimSpec.resources.requests.storage	The size of the disks where Cassandra data will be stored	40Gi

CODE

# install the operator
helm upgrade --install --no-hooks -n <NAMESPACE> -f values/k8ssandra-operator-values.yaml k8ssandra-operator-<ENV_NO> helm/k8ssandra-operator-0.39.2.tgz
# confirm operator is running: wait until status is "Running" and Ready is "1/1"
oc -n <NAMESPACE> get pod | grep k8ssandra-operator
# create the actual cluster
helm upgrade --install -n <NAMESPACE> -f values/k8ssandra-values.yaml k8ssandra-<ENV_NO> helm/vid-k8ssandra-0.5.12.tgz

In case you need to restart the Cassandra pods, use the following command:
kubectl rollout restart statefulset cassandra-<ENV_NO>-dc1-default-sts

5.2. Elasticsearch

The default settings for Elasticsearch are displayed below. If required, they can be modified by editing values/elasticsearch-values.yaml.

Property	Description	Recommended value
elasticsearch.storageClassName	The type of storage to be provisioned for storing Elasticsearch data	<STORAGECLASS_PERSISTENCE>
elasticsearch.diskSize	The size of the disks where Elasticsearch data will be stored	20Gi
elasticsearch.replicaCount	Number of Elasticsearch nodes in cluster	3
Next, execute the following:

CODE


# install the operator
helm upgrade --install -n <NAMESPACE> -f values/eck-operator-values.yaml eck-operator-<ENV_NO> helm/eck-operator-2.1.0.tgz
# confirm operator is running: wait until status is "Running" and Ready is "1/1"
oc -n <NAMESPACE> get pod | grep eck-operator
# create the cluster
helm upgrade --install -n <NAMESPACE> -f values/elasticsearch-values.yaml elasticsearch-<ENV_NO> helm/elasticsearch-0.2.2.tgz
# check elasticsearch
oc -n <NAMESPACE> get pod | grep elasticsearch

5.3. Zookeeper

The default settings for Zookeeper are displayed below. If required, they can be modified by editing the values/zookeeper-values.yaml file.

Property	Description	Recommended value
persistence.storageClassName	The type of storage to be provisioned for storing Zookeeper data	<STORAGECLASS_PERSISTENCE>
persistence.volumeSize	The size of the disks where Zookeeper data will be stored	1Gi
persistence.reclaimPolicy	The reclaim Policy of the disks where Zookeeper data will be stored	Retain
replicas	Number of Zookeeper nodes in cluster	3

CODE

# install the operator
helm upgrade --install -n <NAMESPACE> -f values/zookeeper-operator-values.yaml zookeeper-operator-<ENV_NO> helm/zookeeper-operator-0.2.15.tgz
# create the actual cluster
helm upgrade --install -n <NAMESPACE> --timeout 30m -f values/zookeeper-values.yaml zookeeper-<ENV_NO> helm/zookeeper-0.2.15.tgz
# validate:
oc -n <NAMESPACE> get sts/zookeeper-<ENV_NO>

6. Configure the veridium environment by running the vid-installer chart

6.1 Set configuration variables

The installation can be configured by editing the values/vid-installer-values.yaml file and setting values according to your environment and specific requirements.
Here is a summary of variables available:

Variable	Description
imagePullSecrets.name	Secret containing credentials used for pulling images from private docker registry
cassandraHost	Comma-separated list of hostname(s) or IP(s) of Cassandra. This value is constructed by appending the following values from k8ssandra-deployment.yaml: "`metadata.name`-`spec.cassandra.datacenters[].metadata.name`-service"
zookeeperHost	Comma-separated list of hostname(s) or IP(s) of Zookeeper. Should be: "<NAMESPACE>-zookeeper-client"
elasticsearchHost	Comma-separated list of hostname(s) or IP(s) of Elasticsearch. Should be: "<NAMESPACE>-elasticsearch-es-http"
elasticsearchUser	Secret name where Elasticsearch keeps the username/password. Should be: "elasticsearch-<ENV_NO>-es-elastic-user"
cassandraDatacenter	Name of datacenter as defined in the Cassandra cluster (default "dc1")
cassandraReplicationType	Replication type as defined in the Cassandra cluster (default "NETWORK")
cassandraReplicationFactor	Replication factor as defined in the Cassandra cluster (default 3)
cassandraConsistencyLevel	Consistency leve as defined in the Cassandra cluster (default "LOCAL_QUORUM")
domainName	Domain where this installation will be accessible
environmentName	Name of this installation. This will be the subdomain. So this installation will expect ingress traffic at "environmentName.domainName"
sniEnabled	If true, the services will be exposed under dedicated server names, (e.g. admin-my-env.domain.com, ssp-my-env.domain.com) If false, the services will be exposed under a single name (e.g. my-env.domain.com), but dedicated ports
ssl.generateCert	If true, a self-signed certificate will be generated if false, the defaultCert and defaultKey will be used
ssl.defaultCert	base64 encoded string of a PEM certificate. To generate this string execute: `base64 -w 0 <FULL_PATH_TO_PEM_CERTIFICATE>`
ssl.defaultKey	base64 encoded string of a PEM key. To generate this string execute: `base64 -w 0 <FULL_PATH_TO_PEM_KEY>`

6.2 Start the installer

CODE

helm install -n <NAMESPACE> --timeout 60m -f values/vid-installer-values.yaml vid-installer helm/vid-installer-0.5.12.tgz

This will start the installation process.
Please allow up to 60 minutes to complete.

To monitor the progress, you can use the following command to tail the logs in a separate terminal window:

CODE

oc logs -n <NAMESPACE> --follow $(oc get pod -n <NAMESPACE> --no-headers -o custom-columns=":metadata.name" | grep vid-installer | grep -v vid-installer-post)

7. Deploy webapps

7.1. Configuration of each subsystem

The values/veridiumid-values.yaml file contains configuration properties for each subsystem of VeridiumID. Set the following properties as needed:

Variable	Description
logrotate_conf.*.size	The size in MB for the app log when will be rotated
logrotate_conf.*.rotate	How many copy of log rotation to keep
*.replicaCount	Number of pods for that particular subsystem. This can be easily changed at a later time. If you don't need a particular subsystem, set the replicaCount to 0. (Default value: 2)
vid-haproxy.service.nodePort	The designated port for HAProxy to listen on. This will need to be within the 30000-32767 range, considering it is being exposed as NodePort.
vid-haproxy.service.additionalNodePorts	For "ports"-type installation, the designated port for each subsystem. Default values are: websecadmin: 32102 ssp: 32103 shibboleth-ext: 32104 shibboleth-int: 32105
vid-shibboleth.sessionTimeout	Expressed in seconds. Default: 5
vid-maintenance.backup.zookeeper.enable	Enable the Zookeeper backup using VeridiumID script
vid-maintenance.backup.zookeeper.keep	How many backups to keep
vid-maintenance.backup.zookeeper.schedule	When the cron will run to perform backup for zookeeper
vid-maintenance.backup.elasticsearch.enable	Enable the ElasticSearch backup using VeridiumID script
vid-maintenance.backup.elasticsearch.keep	How many backups of ElasticSearch to keep
vid-maintenance.backup.elasticsearch.schedule	When the cron will run to perform backup for zookeeper
vid-maintenance.backup.backup.elasticsearch_backup_with_policy.enable	Enable ElasticSearch backup with Policy
vid-maintenance.backup.backup.elasticsearch_backup_with_policy.schedule	When to take Elasticsearch snapshots. For example: cron schedule that triggers every day at noon: 0 0 12 * * ?
vid-maintenance.backup.backup.elasticsearch_backup_with_policy.expire_after	How many days to keep the Elasticsearch snaphots. Be default is 30 days.
vid-maintenance.backup.backup.elasticsearch_backup_with_policy.min_count	Minim Elasticsearch snapshots retention regardless of age. By default is 5.
vid-maintenance.backup.backup.elasticsearch_backup_with_policy.max_count	Maxim Elasticsearch snapshots retention regardless of age. By default is 50.
vid-maintenance.backup.cassandraHost	Comma-separated list of hostname(s) or IP(s) of Cassandra. This value is constructed by appending the following values from k8ssandra-deployment.yaml: "`metadata.name`-`spec.cassandra.datacenters[].metadata.name`-service"
vid-maintenance.backup.zookeeperHost	Comma-separated list of hostname(s) or IP(s) of Zookeeper. Should be: "<NAMESPACE>-zookeeper-client"
vid-maintenance.backup.elasticsearchHost	Comma-separated list of hostname(s) or IP(s) of Elasticsearch. Should be: "<NAMESPACE>-elasticsearch-es-http"
vid-maintenance.backup.elasticsearchUser	Secret name where Elasticsearch keeps the username/password. Should be: "elasticsearch-<ENV_NO>-es-elastic-user"

7.2. Launching the web applications

CODE

helm upgrade --install -n <NAMESPACE> -f values/veridiumid-values.yaml veridiumid helm/veridiumid-0.5.12.tgz

7.3. Enable Cassandra encryption (optional)

First, generate certificates for Cassandra by running the following commands:

CODE

maintenance_pod=$(oc -n <NAMESPACE> get pods -o custom-columns=POD:.metadata.name --no-headers --sort-by=.metadata.creationTimestamp | grep vid-maintenance | tail -1)
oc -n <NAMESPACE> exec $maintenance_pod -- bash -c "bash /opt/veridiumid/migration/bin/migration.sh -d /tmp/zk && mkdir -p /tmp/k8ss_tmp && cp /tmp/zk/config.json /tmp/k8ss_tmp/ && cd /tmp/k8ss_tmp && bash /scripts/generate-cassandra-keystore.sh && mv config.json /tmp/zk/ && bash /opt/veridiumid/migration/bin/migration.sh -u /tmp/zk"

Next, edit the values/k8ssandra-values.yaml and set encryption.enabled to true. Next, update the K8ssandra chart by executing:

CODE

helm upgrade --install -n <NAMESPACE> -f values/k8ssandra-values.yaml k8ssandra-<ENV_NO> helm/vid-k8ssandra-0.5.12.tgz

Next, force-restart the Cassandra pods and wait for them to become ready:

CODE

oc -n <NAMESPACE> delete pod cassandra-<ENV_NO>-dc1-default-sts-0
oc -n <NAMESPACE> delete pod cassandra-<ENV_NO>-dc1-default-sts-1
oc -n <NAMESPACE> delete pod cassandra-<ENV_NO>-dc1-default-sts-2
oc -n <NAMESPACE> rollout status statefulset cassandra-<ENV_NO>-dc1-default-sts --timeout=600s

Finally, restart Veridium pods:

CODE

oc -n <NAMESPACE> rollout restart deployment vid-adservice vid-fido vid-freeradius vid-opa vid-shibboleth vid-ssp vid-websec vid-websecadmin

8. Update HAProxy certificate

The installation uses a self-signed certificate for HAProxy. We recommend using a trusted certificate for production purposes.

Please use a PEM file that includes the private key, certificate and full CA chain.

In order to update the certificate please execute the following commands:

CODE

# updates the secret that contains the certificate
oc get secret -n <NAMESPACE> vid-haproxy-certs -o yaml | sed "s,$(oc get secret -n <NAMESPACE> vid-haproxy-certs -ojsonpath='{.data.server\.pem}'),$(base64 -w 0 <FULL_PATH_TO_PEM_CERTIFICATE>),g" | oc apply -f -
# restarts HAProxy in order to load the new certificate from the secret
oc rollout restart -n <NAMESPACE> daemonset/vid-haproxy

8.1 Copy the certificate to WebsecAdmin (OPTIONAL)

In case you need WebsecAdmin to use the same certificate as HAProxy (for instance when exposing WebsecAdmin directly withouh proxying through HAProxy), connect to the vid-maintenance pod and execute the following:

CODE

bash /scripts/update-websecadmin-certificate-secret.sh

9. Configure websecadmin

Please follow the configuration wizard that is presented when first launching WebsecAdmin. Additional instructions can be found at this link.
Once the configuration is complete, please go to Settings - Certificates - Truststores, click on Add Truststore and upload the <FULL_PATH_TO_PEM_CERTIFICATE> file.
To configure Veridium as identity provider, please go to Settings - Services - SSP, click on Configure Veridium as IDP in the right-side panel.

Uninstall procedure

CODE

#
# WARNING: THIS ACTION IS IRREVERSIBLE AND RESULTS IN LOSS OF DATA !
# delete Helm managed resources"
helm -n <NAMESPACE> ls --short | xargs -L1 helm -n <NAMESPACE> delete
# delete Helm-managed resources from 'default' namespace (skip this if the cert-manager was not deployed using veridiumid)
helm delete cert-manager
# delete secrets from 'default' namespace (skip this if cert-manager was not deployed using veridiumid)
oc delete secret vid-credentials-ecr -n default
oc delete secret cert-manager-webhook-ca
# delete the namespace
oc delete -f values/namespace.yaml
# delete Custom Resource Definitions (skip this if CRDs have not been installed using veridiumid)
oc delete -f values/veridiumid-crds/eck-operator-crds.yaml
oc delete -f values/veridiumid-crds/k8ssandra-operator.yaml
oc delete -f values/veridiumid-crds/zookeeper.yaml

Known issues

On some occasions an error similar to the following could be encountered:
"Internal error occurred: failed calling webhook "kb.io - This is a premium name ": Post "https://k8ssandra-operator-<ENV_NO>-cass-operator-webhook-service.<NAMESPACE>.svc:443/validate-cassandra-datastax-com-v1beta1-cassandradatacenter?timeout=10s": Address is not allowed"
To fix it, run the following command then retry:

CODE

oc delete ValidatingWebhookConfiguration k8ssandra-operator-<ENV_NO>-cass-operator-validating-webhook-configuration k8ssandra-operator-<ENV_NO>-validating-webhook-configuration

Stuck in deleting CRD (oc get crd -oname | grep K8ssandra | xargs oc delete)

CODE

oc get crd | grep k8ssandra
oc patch crd/k8ssandraclusters.k8ssandra.io -p '{"metadata":{"finalizers":[]}}' --type=merge

Elasticsearch crashes with the following error: "Unable to access 'path.repo' (/mnt/backups/elasticsearch-backup)"
Make sure the folder exists and is accessible for read/write, as described in 1.2.e.

Maintenance operations

Changing HAProxy configuration

HAProxy configuration is stored in a configmap named vid-haproxy-config in the deployment namespace.

CODE

# edit the configuration
oc edit configmap -n <NAMESPACE> vid-haproxy-config
# restart HAProxy
oc rollout restart -n <NAMESPACE> daemonset/vid-haproxy

Restarting a malfunctioning pod

Most pods have liveness probes in place that cause automatic restart in case of malfunction.
However, if a manual restart is necessary, please execute the following command:

CODE

oc rollout restart -n <NAMESPACE> deployment/<NAME_OF_DEPLOYMENT>