Add manually Zookeeper Datacenter to an existing CDCR

This article will provide a step by step procedure for adding 2 DC’s to an existing DC.

On all persistence nodes (in all datacenters) run the following (one node at a time):

stop webapp services in DC2 and DC3

CODE

bash /etc/veridiumid/scripts/veridium_services.sh stop

stop zookeeper only in DC2 and DC3 and remove old content only on non main DC.
CODE
```
systemctl stop ver_zookeeper
rm -rf /opt/veridiumid/zookeeper/data/version-2
```
reconfigure zookeeper in DC1, DC2, DC3, by running below commands, on each persistence node, one node at a time.

BASH

# Create list of IP Addresses of all persistence nodes (starting with the primary datacenter)
DC1_ZOO=( "IP1_DC1" "IP2_DC1" "IP3_DC1" )
DC2_ZOO=( "IP1_DC2" "IP2_DC2" "IP3_DC2" )
DC3_ZOO=( "IP1_DC3" "IP2_DC3" "IP3_DC3" )

# Stop Zookeeper
systemctl stop ver_zookeeper

# Make copy of current configuration file
\cp /etc/veridiumid/zookeeper/zoo.cfg /etc/veridiumid/zookeeper/zoo.cfg_before_cdcr

# Make backup of myid file
\cp /opt/veridiumid/zookeeper/data/myid /opt/veridiumid/zookeeper/data/myid_before_cdcr

# Get current ID and IP of node
current_id=`cat /opt/veridiumid/zookeeper/data/myid | tr -d [:space:]`
current_ip=`cat /opt/veridiumid/zookeeper/conf/zoo.cfg | grep "server.${current_id}" | cut -d"=" -f2 | cut -d":" -f1 | tr -d [:space:]`

# Remove current server configs from zoo.cfg
sed -i '/^server.*/d' /opt/veridiumid/zookeeper/conf/zoo.cfg

# Add configurations
grep -q "standaloneEnabled=false" /opt/veridiumid/zookeeper/conf/zoo.cfg; if [ $? -ne 0 ]; then echo -ne "standaloneEnabled=false\n" >> /opt/veridiumid/zookeeper/conf/zoo.cfg; fi
grep -q "localSessionsEnabled=true" /opt/veridiumid/zookeeper/conf/zoo.cfg; if [ $? -ne 0 ]; then echo -ne "localSessionsEnabled=true\n" >> /opt/veridiumid/zookeeper/conf/zoo.cfg; fi
grep -q "localSessionsUpgradingEnabled=true" /opt/veridiumid/zookeeper/conf/zoo.cfg; if [ $? -ne 0 ]; then echo -ne "localSessionsUpgradingEnabled=true\n" >> /opt/veridiumid/zookeeper/conf/zoo.cfg; fi
echo "" >> /opt/veridiumid/zookeeper/conf/zoo.cfg

new_id=0
for ip_addr in ${DC1_ZOO[@]};
do
	new_id=$((new_id+1))
	echo -ne "server.${new_id}=${ip_addr}:2888:3888:participant;${ip_addr}:2181\n" >> /opt/veridiumid/zookeeper/conf/zoo.cfg
	if [[ "${ip_addr}" == "${current_ip}" ]];
	then
		echo "${new_id}" > /opt/veridiumid/zookeeper/data/myid
	fi
done
for ip_addr in ${DC2_ZOO[@]};
do
	new_id=$((new_id+1))
    echo -ne "server.${new_id}=${ip_addr}:2888:3888:observer;${ip_addr}:2181\n" >> /opt/veridiumid/zookeeper/conf/zoo.cfg
	if [[ "${ip_addr}" == "${current_ip}" ]];
	then
		echo "${new_id}" > /opt/veridiumid/zookeeper/data/myid
	fi
done

for ip_addr in ${DC3_ZOO[@]};
do
	new_id=$((new_id+1))
    echo -ne "server.${new_id}=${ip_addr}:2888:3888:observer;${ip_addr}:2181\n" >> /opt/veridiumid/zookeeper/conf/zoo.cfg
	if [[ "${ip_addr}" == "${current_ip}" ]];
	then
		echo "${new_id}" > /opt/veridiumid/zookeeper/data/myid
	fi
done
# Start zookeeper
systemctl start ver_zookeeper
check_services

On all nodes in the DC2 and DC3 adjust zookeeper.properties file, as on first node.

First get the configurations from DC1, by runnning this on one node in DC1 (webapp or persistence)

BASH

zoo_user=`cat /etc/veridiumid/zookeeper.properties | grep username | cut -d"=" -f2 | tr -d [:space:]`
zoo_pass=`cat /etc/veridiumid/zookeeper.properties | grep password | cut -d"=" -f2 | tr -d [:space:]`
zoo_salt=`cat /etc/veridiumid/zookeeper.properties | grep salt | cut -d"=" -f2 | tr -d [:space:]`
echo "zoo_user=$zoo_user"
echo "zoo_pass=$zoo_pass"
echo "zoo_salt=$zoo_salt"

take the ouput of previous command and run it on all nodes togather with below sed’s:

BASH

zoo_user=....
zoo_pass=....
zoo_salt=....
\cp /etc/veridiumid/zookeeper.properties /etc/veridiumid/zookeeper.properties_before_cdcr
sed -i "s|:2181/.*/|:2181/$zoo_user/|g" /etc/veridiumid/zookeeper.properties
sed -i "s|username=.*|username=${zoo_user}|g" /etc/veridiumid/zookeeper.properties
sed -i "s|password=.*|password=${zoo_pass}|g" /etc/veridiumid/zookeeper.properties
sed -i "s|salt=.*|salt=${zoo_salt}|g" /etc/veridiumid/zookeeper.properties

Start webapp services in DC2 and DC3

BASH

# Start all services
bash /etc/veridiumid/scripts/veridium_services.sh start