VeridiumID Server On-Prem - Cross DataCenter Replication using the same CA (VeridiumID 3.8.4+)

This article will provide a step by step procedure to create a Cross Datacenter Configuration between VeridiumID deployments.

1. Pre-reqs

Before starting the CDCR configuration make sure that you have a backup of Zookeeper configurations on both datacenters.

To generate a backup run the following commands on any node in both datacenters:

sudo bash /opt/veridiumid/migration/bin/migration.sh -d ZOO_BKP

Where ZOO_BKP is the name of the directory where the configurations will be stored

1.1) User requirements

The procedure must be executed using root or a user with sudo privileges.

1.2) Network connectivity

The following ports should be opened between persistence nodes in both datacenters:

zookeeper: 2888, 3888,
cassandra: 7001, 9042, 7000
elasticsearch: 9092, 9095

In order to test connectivity:

CODE

## test if a port is opened on a machine
netstat -tulpn | grep 2888
## test from other machine the connectivity to that port
nc -zv IP 2888

1.3) Cassandra RPC port configuration (MANDATORY!!)

Make sure the Cassandra is using port 7001 as RPC port (uses for communicating between nodes).

BASH

# Run the following commands to check which RPC port is used
grep "seeds:" /etc/veridiumid/cassandra/cassandra.yaml

# Expected result
# - seeds: "1.2.3.4:7001,1.2.3.5:7001,1.2.3.6:7001"

grep "^storage_port" /etc/veridiumid/cassandra/cassandra.yaml

# Expected result
# storage_port: 7001

In case it is using port 7000 the following commands must be used in order to modify it:

This operation will also restart Cassandra.

This operation must be performed by root or a user with sudo privileges.

BASH

# Change seed list ports
sed -i "s|\:7001|\:7000|g" /etc/veridiumid/cassandra/cassandra.yaml

# Change storage port
sed -i "s|^storage_port.*|storage_port\: 7001|g" /etc/veridiumid/cassandra/cassandra.yaml

# Restart Cassandra service
systemctl restart ver_cassandra

2) Initiate the CDCR configuration

2.1) Stop all services on all nodes in the new datacenter (except for ver_cassandra)

To stop all services run the following command on all nodes in the new datacenter:

BASH

sudo bash /etc/veridiumid/scripts/veridium_services.sh stop

# On persistence nodes make sure to start Cassandra service
service ver_cassandra start

2.2) Create an archive containing all necessary information for the CDCR configuration

Connect to a Webapp node in the primary datacenter and run the following command (primary datacenter can be identified by checking the zookeeper node to be in leader state):

BASH

sudo bash /etc/veridiumid/scripts/veridiumid_cdcr.sh -i

The command above will gather all required information regarding the new datacenter that will be added, including the IP addresses of the webapp and persistence nodes of the new datacenter and the new datacenter name.

The following details will be requested:

WEBAPP IP addresses: enter all the IP addresses of the webapp nodes in the new datacenter, separated by comma. e.g. 10.204.90.15,10.204.90.16,10.204.90.17
PERSISTENCE IP addresses: enter all the IP addresses of the persistence nodes in the new datacenter, separated by comma. e.g. 10.204.90.15,10.204.90.16,10.204.90.17
NEW DATACENTER NAME: To get the new datacenter name you can connect to a persistence node in the new datacenter and run the following command. In the example below, the datacenter name is dc1 :

CODE

sudo /opt/veridiumid/cassandra/bin/nodetool status
# Expected output:
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load        Tokens  Owns (effective)  Host ID                               Rack 
UN  10.204.0.107  324.39 MiB  8       100.0%            f32a2a58-6e3e-4b68-a948-2df1bb499899  rack1

The script will:

generate the following archive: /tmp/DC1.tar.gz, containting zookeeper content and necessary data for freeradius and haproxy (server pems).
also it updates zookeper content in current datacenter to include the new nodes → tenant-infra.yaml, cassandra connection (config.json) and elasticsearch connection (elasticsearch.json); modify zoo.cfg template from zookeeper.

Copy the generated archive on all nodes in the new datacenter (both persistence and webapp nodes) and on all persistence nodes in the primary (existing) datacenter.

The command used above to generate the CDCR archive can also be used in non-interactive mode. To run it without being asked for information please use the following sintax:

BASH

sudo bash /etc/veridiumid/scripts/veridiumid_cdcr.sh -i -w 10.1.1.1,10.1.1.2 -s 10.2.2.1,10.2.2.2,10.2.2.3 -n newdc2

# Parameters used:
#  -w -> list of webapp nodes delimited by commas
#  -p -> list of persistence nodes delimited by commas
#  -n -> the new datacenter's name

3. Configure the new datacenter

The veridiumid_cdcr.sh script has also the option to skip configuring Cassandra or ElasticSearch (in case of using Oracle DB or external ElasticSearch).

To skip configuring them use the following parameters alongside the commands used in this section:

-c To skip configuring Cassandra

-e To skip configuring ElasticSearch

3.1) Configure Persistence nodes

Run the following command on ALL (new and old) persistence nodes (one at a time), starting with the ones in the new datacenter:

BASH

sudo bash /etc/veridiumid/scripts/veridiumid_cdcr.sh -a FULL_PATH_TO_DC1.tar.gz

This command will perform the following:

Configure the new datacenter for the following services: Cassandra, Zookeeper, ElasticSearch
Configure Filebeat and SetupAgent connections:
- Configure truststores
- Modify zookeeper.properties
Restart all persistence services

After finishing all persistence nodes (in both datacenters) run this command on ALL nodes in the new datacenter (one at a time).

BASH

sudo bash /etc/veridiumid/scripts/veridiumid_cdcr.sh -p

This command will perform the following:

Prompt Cassandra to start replicating the data in the primary datacenter
Configure the new values for shards and replicas for existing ElasticSearch indexes

3.2) Configure Webapp nodes

Run the following command on all webapp nodes in the new datacenter (one at a time):

CODE

sudo bash /etc/veridiumid/scripts/veridiumid_cdcr.sh -a FULL_PATH_TO_DC1.tar.gz

This command will perform the following:

Synchronize zookeeper.properties file
Configure the new truststore for Webapp services
Restart all webapp services

In case the new datacenter was deployed using a different FQDN than the primary datacenter, make sure to configure the primary datacenter’s FQDN in /etc/hosts

4. Add more datacenters

In case of adding more datacenters the following command must be executed on all nodes from the current configured datacenters.

BASH

sudo bash /etc/veridiumid/scripts/veridiumid_cdcr.sh -d

This command removes the state files configured by the Veridium CDCR configuration script for each service that was modified.

After performing the above command on all existing nodes from the current configured datacenters, please run sections 1, 2 and 3 in order to configure the new datacenter.

5. Rollback operation

In case of issues during initial phases of the procedure run the following command to rollback the Zookeeper configuration:

CODE

sudo bash /opt/veridiumid/migration/bin/migration.sh -u /etc/veridiumid/scripts/CURRENT_CONFIG

And afterwards delete the state files created from adding a third datacenter (if they exist) (see section 4) and start from the beginning.