Configure a different certificate for Cassandra, Elasticsearch, Filebeat and Kibana

Cassandra certificate should have the following properties:

Client Authentication and Server Authentication as usage
Alternatives names, all the IP’s of the servers.

There are several way to get this kind of certificate. Here are 2 of them:

Use the following CSR, as an example. Please modify the CN, alt_names, accordingly.

CODE

[ req ]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
req_extensions = req_ext
prompt = no

[ req_distinguished_name ]
C   = RO
ST  = RO
L   = Bucharest
O   = VeridiumID
OU  = Veridium
CN  = cassandra.dev.local

[ v3_req ]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth

[req_ext]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = 10.204.60.1
DNS.2 = 10.204.60.2
DNS.3 = 10.204.60.3
DNS.4 = 10.204.60.4
DNS.5 = 10.204.60.5
DNS.6 = 10.204.60.6
DNS.7 = cassandra.dev.local

Generate a private key and a certificate request, that needs to be signed by an external authority. Take file cassandra.csr and sign the certificate.

CODE

openssl genrsa -out cassandra.key 2048
openssl req -new -key cassandra.key -out cassandra.csr -config cassandra.cnf

If Pfx is provided, containing the Private key, certificate and chain, the following needs to be executed:

CODE

convert_haproxy_cert.sh file.pfx

Having the certificate, now it needs to be configured for the processes:

Check if certificate is correct:

CODE

## check that the certifiate is correctly generated. It should be Server and Client Authentication.
openssl x509 -in server.pem -text -noout | grep "Extended Key Usage" -A4
#            X509v3 Extended Key Usage:
#                TLS Web Server Authentication, TLS Web Client Authentication

Import the intermediate certificate in both Veridium and ILP truststore:

CODE

##
in websecadmin -> add certificate to Truststore

in ILP -> add certificate on each node, by running the following commands:

passkeystore=`grep CASSANDRA_KEYSTORE_PASS_ENV /etc/default/veridiumid/uba_variables | awk -F'"' '{print $2}'`
keytool -importcert -keystore /opt/veridiumid/uba/TrustStore.jks -storepass $passkeystore -file intermediate.pem
cat intermediate.pem >> /opt/veridiumid/uba/TrustStore.pem

cat /etc/default/veridiumid/uba_variables | grep TRUSTSTORE_ENV | awk -F'\"' '{print $2}' | base64 -d > cert_ilp.pem
cat intermediate.pem >> cert_ilp.pem
sed -i "/TRUSTSTORE_ENV=(.*)/d" /etc/default/veridiumid/uba_variables
echo "TRUSTSTORE_ENV=\""$(base64 -w0 cert_ilp.pem)\" >> /etc/default/veridiumid/uba_variables

Cassandra

Once the certificate is signed, it needs to be changed for Cassandra (on each cassandra node). It is necessary to have also the intermediate chain for the certificate in a separate file.

CODE

cat cassandra.key > server.pem
cat cassandraSigned.pem >> server.pem
cat intermediate.pem >> server.pem

CURRENT_TIMESTAMP=`date +"%Y%m%d%H%M%S%3N"`
CAS_KEYSTORE=/opt/veridiumid/cassandra/conf/KeyStore.jks
CAS_TRUSTSTORE=/opt/veridiumid/cassandra/conf/TrustStore.jks
CAS_PASS=$(grep "truststore_password" /opt/veridiumid/cassandra/conf/cassandra.yaml | awk -F' ' '{print $2}' | sort -u)
##backup
\cp $CAS_KEYSTORE $CAS_KEYSTORE.$CURRENT_TIMESTAMP
\cp $CAS_TRUSTSTORE $CAS_TRUSTSTORE.$CURRENT_TIMESTAMP
##copy locally and process them
openssl pkcs12 -export -in server.pem -inkey server.pem -name 'tomcat' -out KeyStore.jks -passout pass:${passCass}
cp $CAS_TRUSTSTORE TrustStore.jks
echo "yes" | keytool -import -alias "cassandra" -file intermediate.pem -storetype JKS -keystore TrustStore.jks -storepass ${CAS_PASS}

###transfor JKS to PKCS12 for cassandra. It is safer to have it like this.
keytool -importkeystore -srckeystore KeyStore.jks -destkeystore KeyStore.jks -srcstoretype jks -deststoretype pkcs12 -srcstorepass ${CAS_PASS} -keypass ${CAS_PASS} -deststorepass ${CAS_PASS} -noprompt
keytool -importkeystore -srckeystore TrustStore.jks -destkeystore TrustStore.jks -srcstoretype jks -deststoretype pkcs12 -srcstorepass ${CAS_PASS} -keypass ${CAS_PASS} -deststorepass ${CAS_PASS} -noprompt

## in order to avoid downtime, modify the truststore first, do rolling restart on all 6 nodes
\cp TrustStore.jks ${CAS_TRUSTSTORE}
systemctl restart ver_cassandra

## and after that modify the keystore and do rolling restart
\cp KeyStore.jks $CAS_KEYSTORE
systemctl restart ver_cassandra

Elasticsearch

CODE

## copy keystore and truststore from Cassandra

CURRENT_TIMESTAMP=`date +"%Y%m%d%H%M%S%3N"`
CAS_KEYSTORE=/opt/veridiumid/cassandra/conf/KeyStore.jks
CAS_TRUSTSTORE=/opt/veridiumid/cassandra/conf/TrustStore.jks
CAS_PASS=$(grep "truststore_password" /opt/veridiumid/cassandra/conf/cassandra.yaml | awk -F' ' '{print $2}' | sort -u)

ELK_KEYSTORE=/opt/veridiumid/elasticsearch/config/certs/KeyStore.p12
ELK_TRUSTSTORE=/opt/veridiumid/elasticsearch/config/certs/TrustStore.p12
ELK_PASS=$(/opt/veridiumid/elasticsearch/bin/elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password)

###create backup
  \cp $CAS_KEYSTORE $CAS_KEYSTORE.$CURRENT_TIMESTAMP
  \cp $CAS_TRUSTSTORE $CAS_TRUSTSTORE.$CURRENT_TIMESTAMP
  \cp $ELK_KEYSTORE $ELK_KEYSTORE.$CURRENT_TIMESTAMP
  \cp $ELK_TRUSTSTORE $ELK_TRUSTSTORE.$CURRENT_TIMESTAMP

### copy keystore and truststore and adjust the password
  \cp $CAS_KEYSTORE $ELK_KEYSTORE
  \cp $CAS_TRUSTSTORE $ELK_TRUSTSTORE

  keytool -storepasswd -new $ELK_PASS -keystore $ELK_KEYSTORE -storepass $CAS_PASS
  keytool -storepasswd -new $ELK_PASS -keystore $ELK_TRUSTSTORE -storepass $CAS_PASS

disable: eops -x=PUT -p=/_cluster/settings -d="{\"transient\":{\"cluster.routing.allocation.enable\":\"none\"}}"

systemctl restart ver_elasticsearch

enable: eops -x=PUT -p=/_cluster/settings -d="{\"transient\":{\"cluster.routing.allocation.enable\":\"all\"}}"

Filebeat and Kibana

CODE

### take the ca from Elastic, and put it to kibana and filebeat. This needs to be executed also on WEBAPP, for KIBANA and FILEBEAT. On persitence needs to be executed for FILEBEAT.
## so first extract the truststore on persistence node and copy file /tmp/all-certs.pem to webapp.
CAS_KEYSTORE=/opt/veridiumid/cassandra/conf/KeyStore.jks
CAS_TRUSTSTORE=/opt/veridiumid/cassandra/conf/TrustStore.jks
CAS_PASS=$(grep "truststore_password" /opt/veridiumid/cassandra/conf/cassandra.yaml | awk -F' ' '{print $2}' | sort -u)

keytool -list -rfc -keystore $CAS_TRUSTSTORE -storepass $CAS_PASS | awk '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/' > /tmp/all-certs.pem
less /tmp/all-certs.pem

##!!!! It might be missing one of the certificates from the chain (it should be both intermediate and root), so you need to add it manually to ca.pem!!! Otherwise the kibana and filebeat will not work

\cp /tmp/all-certs.pem /opt/veridiumid/kibana/config/certs/ca.pem
\cp /tmp/all-certs.pem /opt/veridiumid/filebeat/config/certs/ca.pem

systemctl restart ver_kibana
systemctl restart ver_filebeat