Decommission one DC from CDCR

1) stop services in the DC that is going to be removed on all nodes, one by one, first on webapp and later on persistence (DC1)

CODE

ver_stop

2) on remaining DC (DC2) Elasticsearch - remove old nodes

CODE

## on a persistence node run the following
eops -l
## set number of replicas to one
eops -x=PUT -p=/veridium.*/_settings -d='{"index":{"number_of_replicas":1}}'
## on all nodes do:
vi /opt/veridiumid/elasticsearch/config/elasticsearch.yml
## remove the olde nodes from discovery.seed_hosts
## comment cluste.initial_master_nodes
## comment cluster.routing.allocation.awarness.force.zone.values
restart elasticsearch on all nodes - one after the other!!
systemctl restart ver_elasticsearch.service
check_services
# --> should be green
##run eops -l to verify the replica status

3) on remaining DC (DC2) Cassandra - remove old nodes

CODE

## get the HOST ID that needs to be removed:
nodetool status
nodetool removenode HOSTID
## in the end, it should remain in the cluster only the nodes from current datacenter
##modify cassandra.yaml
vi /opt/veridiumid/cassandra/conf/cassandra.yaml
## seeds -> set only 2 nodes from current datacenter
## update topology
/opt/veridiumid/cassandra/bin/cqlsh --ssl --cqlshrc=/opt/veridiumid/cassandra/conf/veridiumid_cqlshrc
# ACPT
ALTER KEYSPACE system_auth WITH REPLICATION = {'class' :'NetworkTopologyStrategy', 'dc2' : 3};
ALTER KEYSPACE system_traces WITH REPLICATION = {'class' :'NetworkTopologyStrategy', 'dc2' : 3};
ALTER KEYSPACE system_distributed WITH REPLICATION = {'class' :'NetworkTopologyStrategy', 'dc2' : 3};
ALTER KEYSPACE veridium WITH REPLICATION = {'class' : 'NetworkTopologyStrategy','dc2': 3};
quit
##restart cassandra on all ndoes - one after the other
systemctl restart ver_cassandra.service
nodetool status
nodetool describecluster
##Run full repair on new persistence node 1
/opt/veridiumid/cassandra/bin/nodetool repair --full