Operational runbook

This article will provide a list with all the troubleshooting tools and scripts present on deployed environments.

Disk path	Script Name	Node Type (webapp/persistence)
/etc/veridiumid/scripts/	backup_configs.sh	All
	config_revert.sh	All
	convert_haproxy_cert.sh	webapp
	manage_alerts.py	All
	check_license.py	Webapp
	check_prepreqs.sh	All
	check_services.sh	All
	server_pins.sh	Webapp
	veridium_services.sh	All
	check_certificates.py	All
	check_domain_cert.sh	Webapp
	getLogs.sh	All
	gather_env_info.sh	All
	campain_results.sh	Persistence
	znode_cleanup.py	Webapp
	check_db_ops.sh	Persistence
/opt/veridiumid/elasticsearch/bin/	elasticsearch_ops.sh	Persistence
where the veridium installation is uncompressed or in /home/veridiumid	check_prereqs.sh	InstallationNode
/etc/veridiumid/update-procedure/current	update_manager.sh	All

CLI Command - Alias	Full comand	Node (webapp/persistence)
check_services	bash /etc/veridiumid/scripts/check_services.sh	all
ver_stop	bash /etc/veridiumid/scripts/veridium_services.sh stop	all
ver_start	bash /etc/veridiumid/scripts/veridium_services.sh start	all
ver_disable	bash /etc/veridiumid/scripts/veridium_services.sh disable	all
ver_enable	bash /etc/veridiumid/scripts/veridium_services.sh enable	all
check_certificates	python3 /etc/veridiumid/scripts/check_certificates.py	persistence
cqlsh	/opt/veridiumid/cassandra/bin/cqlsh --cqlshrc=/opt/veridiumid/cassandra/conf/veridiumid_cqlshrc --ssl	persistence
zkcli	/opt/veridiumid/zookeeper/bin/zkCli.sh	persistence
ver_getLogs	bash /etc/veridiumid/scripts/getLogs.sh	all
eops	bash /opt/veridiumid/elasticsearch/bin/elasticsearch_ops.sh

All scripts must be executed as root user.

1. backup_configs.sh

The backup configs script will perform a backup of all required configurations present on disk on each node, including the JSON configuration files.

Usage:

CODE

bash /etc/veridiumid/scripts/backup_configs.sh /etc/veridiumid/scripts/backup_configs.conf

2. config_revert.sh

The script will recover the node using the configurations taken from a configuration backup (the one done by backup_configs.sh)

Usage:

CODE

Usage: ./config_revert.sh <ARGS>
Args:                     -c CONFIG_FILE     - full path to the configuration file (same one used for the config backup)
                          -b BACKUP_ZIP      - full path to the bakcup archive
                          -t TRANSITION_FILE - full path to the transition file (containing IP addresses transition)
                          -j                 - run just the JSON upload

Local config revert:
  bash /etc/veridiumid/scripts/config_revert.sh -c /etc/veridiumid/scripts/backup_configs.conf -b /opt/veridiumid/backup/all_configs/BACKUP_ARCHIVE_NAME -t PATH_TO_TRANSITION_FILE
  
JSON Upload:
  bash /etc/veridiumid/scripts/config_revert.sh -c /etc/veridiumid/scripts/backup_configs.conf -b /opt/veridiumid/backup/all_configs/BACKUP_ARCHIVE_NAME -t PATH_TO_TRANSITION_FILE -j
  
Example Transition file:
OLD_IP1:NEW_IP1
OLD_IP2:NEW_IP2
OLD_IP3:NEW_IP3

3. convert_haproxy_cert.sh

The script will convert from a PKCS12 certificate to the server.pem file required by HaProxy service.

CODE

Usage: ./convert_haproxy_cert.sh PATH_TO_PKCS_FILE

Example: 

bash /etc/veridiumid/scripts/convert_haproxy_cert.sh /home/veridiumid/veridium.p12

4. manage_alerts.py

The below script is triggering specific alerts.

CODE

Usage: python3 /etc/veridiumid/alerts/manage_alerts.py --config /etc/veridiumid/alerts.conf

##The following alerts should be enabled in crontab.
Please create 2 files and setup in crontab to run as follow:

28 15 * * * python3 /etc/veridiumid/alerts/manage_alerts.py --config /etc/veridiumid/alertsDaily.conf

/etc/veridiumid/alertsDaily.conf
##Daily on one webapp and on one persistence
check_certificates.py:/etc/veridiumid/:ENABLE:/usr/bin/python3:warn:raep:toUser@domain.com:Check certificates results

*/10 * * * * python3 /etc/veridiumid/alerts/manage_alerts.py --config /etc/veridiumid/alertsEvery10Min.conf
/etc/veridiumid/alertsEvery10Min.conf
##Every 10 minutes, on each webapp and each persistance:
check_services.sh:/etc/veridiumid/scripts/:ENABLE:/bin/bash:starting,stopped:NONE:toUser@domain.com:Check services result
checkLdapConnections.sh:/etc/veridiumid/scripts/:ENABLE:/bin/bash:ERROR,RECOVERED:NONE:toUser@domain.com:LDAP connection status has changed
resource:disk_check:ENABLE:5:toUser@domain.com:Check disk usage
resource:mem_check:ENABLE:5:toUser@domain.com:Check memory usage

5. check_license.py

The check license script is used to validate in the license imported into the VeridiumID server is valid (has not expired and the server pinning provided is the correct one).

Usage:

CODE

python3 /etc/veridiumid/scripts/check_license.py

6. check_prereqs.sh

The check pre-requisites script will validate the following:

if the list of required packages present in official repositories (RedHat repositories) are installed
the NTP service is synchronized
the minimum amount of RAM has been allocated to the node

Usage:

CODE

bash /etc/veridiumid/scripts/check_prereqs.sh

7. check_services.sh

The check services script will validate if the correct services are running on the node.

Usage:

CODE

bash /etc/veridiumid/scripts/check_services.sh

or using an alias:

check_services

8. server_pins.sh

The server pins script is used to provide the server pinning of the domain certificate (required for generating a license for VeridiumID services).

Usage:

CODE

With path to HaProxy domain certificate:
  bash /etc/veridiumid/scripts/server_pins.sh /etc/veridiumid/haproxy/server.pem
  
With domain name:
  bash /etc/veridiumid/scripts/server_pins.sh veridium.my-domain.com

9. veridium_services.sh

The veridium services script is used to stop/start/disable/enable all VeridiumID services present on the node.

Usage:

CODE

bash /etc/veridiumid/scripts/veridium_services.sh stop
bash /etc/veridiumid/scripts/veridium_services.sh start
bash /etc/veridiumid/scripts/veridium_services.sh disable
bash /etc/veridiumid/scripts/veridium_services.sh enable

or using aliases:

ver_stop
ver_start
ver_disable
ver_enable

10. check_certificates.py

The check certificates script is used to check the validity of all certificates used in the VeridiumID server (Friend, Default, Admin and device certificates).

The script should be executed on one WEBAPP and one PERSISTENCE in each datacenter. This is necessary, because it takes the information from Zookeeper, Haproxy and also from Cassandra.

Usage:

CODE

python3 /etc/veridiumid/scripts/check_certificates.py

or using an alias:

check_certificates

11. check_domain_cert.sh

The check domain certificate script will validate that all certificates are present in the domain cert set at HaProxy level.

Usage:

CODE

bash /etc/veridiumid/scripts/check_domain_cert.sh /etc/veridiumid/haproxy/server.pem

12. getLogs.sh

This script collect the logs for a specific date in order to be send to Veridium for troubleshoot.

The script will generate an archive with all the Veridium logs, in the location where the script is executed.

Usage:

CODE

#get the logs from today
bash /etc/veridiumid/scripts/getLogs.sh

#get the logs for a specific date, YYYY - year, MM - month, DD - day; example: 20220730
bash /etc/veridiumid/scripts/getLogs.sh YYYYMMDD

#get the logs for a specific intervarl, from first date to second date
bash /etc/veridiumid/scripts/getLogs.sh YYYYMMDD YYYYMMDD

13. gather_env_info.sh

This script will gather data required for better troubleshooting, data regarding:

The memory, CPU and disk allocated to the node
Which VeridiumID service are running on the node and their statuses
How much disk every VeridiumID service is occupying
VeridiumID services startup parameters
Node Kernel and OS version
JAVA version
Sysctl configurations (used for tunning)
Content of the hosts file
Which processes are consuming the most resources (using TOP)

Usage:

CODE

bash /etc/veridiumid/scripts/gather_env_info.sh

14. campain_results.sh

This script will generate a list of users that logged in and did not change PIN
Also it will show the campaign progress (how many users have changed the PIN and how many users has logged on).

Usage:

CODE

bash /etc/veridiumid/reports/campain_results.sh

15. check_prereqs.sh

this script checks pre installation checks, meaning:
- port connectivity with wget and nc
- checks if prerequisites are installed
- checks if ntp is in sync
- checks RAM and disk

Usage:

CODE

Usage: ./check_prereqs.sh <args>
Args:  -l          -> Run on local node 
       -r          -> Run on remote nodes
       -w IPS_LIST -> webapp node ip list delimitted by commas, for example: '10.0.0.1,10.0.0.2'
       -p IPS_LIST -> persistence node ip list delimitted by commas, for example: '10.0.0.1,10.0.0.2'
Examples:
## local server
./check_prereqs.sh -l
## one server, checks remote
./check_prereqs.sh -r -w 10.109.20.133 -p 10.109.20.133
## 2 webapps and 3 persistance
./check_prereqs.sh -r -w 10.109.20.133,10.109.21.164 -p 10.109.50.171,10.109.51.185,10.109.52.72

16. znode_cleanup.py

This script will remove old Zookeeper configuration nodes from previous versions (created from updating environments).

Usage:

CODE

usage: znode_cleanup.py [-h] [--print_znodes] [--znode ZNODE] [--limit LIMIT]
                        [--delete] [--debug]

VeridiumID Zookeeper config read script

optional arguments:
  -h, --help      show this help message and exit
  --print_znodes  Print the existing configuration Zookeeper nodes
  --znode ZNODE   The name of the Zookeeper node that will be deleted
  --limit LIMIT   The number of Zookeeper nodes that should remain after the
                  deletion is done
  --delete        To configure the delete operation
  --debug         If Debug mode is enabled

Examples:
## To get existing Zookeeper configuration nodes:
python3 znode_cleanup.py --print_znodes

Example Output:
[INFO]The available Zookeeper configuration nodes are the following: ['6.1.20', '6.1.22', '6.1.24', '6.1.26', '6.1.27', '6.1.28', '6.1.29', '6.1.35', '6.1.36', '6.2.3', '6.2.19', '6.3.1', '6.4.3']

## Delete a single Zookeeper configuration node using the node's name
python3 znode_cleanup.py --delete --znode 6.1.20

Example Output:
[INFO]Removing /veridiumid/6.1.20

## Delete all but the last N Zookeeper configuration nodes using the limit argument
python3 znode_cleanup.py --delete --limit 10

Example Output:
[INFO]Removing /veridiumid/6.1.20
[INFO]Removing /veridiumid/6.1.22
[INFO]Removing /veridiumid/6.1.24

17. check_db_ops.sh

This script will check if the last Cassandra backup/maintenance task has been done successfully. It can be used in association with manager_alerts.py.

Usage:

CODE

bash /etc/veridiumid/scrips/check_db_ops.sh

Example Ouput:
[INFO] Successful operation found in /var/log/veridiumid/cassandra/backup.log: 20-dec-2022 11:15:35 INFO Finished snapshot: dc1_127.0.0.1
[INFO] Log file /var/log/veridiumid/cassandra/maintenance.log is empty. Checking previous log file...
[INFO] Successful operation found in /var/log/veridiumid/cassandra/maintenance.log-20221220: 20-dec-2022 07:23:34 INFO Nodetool repair finishes successfully

18. update_manager.sh

This is an additional, alternative way to do the upgrade.

The node from where the upgrade will be started must have a user that SSH connectivity using SSH keys to all other deployment nodes.

The remote user must have SUDO permissions or must be the root user.

On the deployment node install the latest veridiumid_update_procedure RPM and from the /etc/veridiumid/update-procedure/current directory run the following command to start the upgrade process:

CODE

# In case of using YUM repositories
bash update_manager.sh -v VERSION

# In case of using local RPM packages
bash update_manager.sh -v VERSION -r RPM_PATH

Where:
  - VERSION -> is the build number of the new VeridiumID version, for example: 6.2.2
  - RPM_PATH -> is the full path to the directory containing all RPMs required by the update process, for example: /home/veridiumid/rpms

19. elasticsearch_ops.sh

Usage:

CODE

Required parameters:
-b, --backup        Runs a backup operation on the ElasticSearch cluster
-r=, --restore=     Runs a restore operation on the ElasticSearch cluster. Requires the name of the snapshot that will be restored
-l, --list          Lists the ElasticSearch snapshots
-i, --indices       List the indices and their settings
-v, --debug         Enable debug logging
-x=, --request=     The request command: GET/PUT/POST/DELETE
-p=, --path=        The API path
-d=, --data=        The data required for the PUT API call
-h, --help          Prints in the standard output the script's usage

To run API calls the following parameters must be used:

-x=/--request=
- To configure the request method
- Available request methods: GET/PUT/POST/DELETE
-p=/--path=
- To configure the API path
- For example: /_cat/repositories
- More details can be found at: ElasticSearch APIs
-d=/--data=
- Optional parameter
- To add the JSON request string

PUT requests require a JSON data string.

Examples:

To check the current indexes states and distribution among all cluster nodes use the following command:
CODE
```
eops -i
```
Check available snapshot repositories
- CODE
```
eops -x=GET -p=/_cat/repositories
```
Delete index
- CODE
```
eops -x=DELETE -p=/REPOSITORY_NAME
```

Modify cluster.routing.allocation.enable value

CODE

eops -x=PUT -p=/_cluster/settings -d='{"persistent":{"cluster.routing.allocation.enable":"primaries"}}'

The JSON request must not contain spaces.