Elastic Search entities index procedure

Devices

Any modification on the device entity generates a commit log entry in the device_history_commit_log table. Right after this save, a new thread attempts to asynchronously index all the commit logs of that specific device, and, if successful, deletes all these entries afterwards.

The CommitLogCleanupJob, runs every 30 seconds and attempts to clean the old entries remaining from the device_history_commit_log. It takes chunks of 100 devices at a time, iterates over each device and reads all commit logs associated with it, and attempts to index them in elastic. If successful, the commit logs of that specific device are deleted.

Accounts

Any modification on the device entity generates a commit log entry in the account_history_commit_log table. Right after this save, a new thread attempts to asynchronously index all the commit logs of that specific account, and, if successful, deletes all these entries afterwards.

The CommitLogCleanupJob, runs every 30 seconds and attempts to clean the old entries remaining from the account_history_commit_log. It takes chunks of 100 accounts at a time, iterates over each account and reads all commit logs associated with it, and attempts to index them in elastic. If successful, the commit logs of that specific account are deleted.

Profiles

Any modification on the profile entity generates a commit log entry in the profile_history_commit_log table. Right after this save, a new thread attempts to asynchronously index all the commit logs of that specific profile, and, if successful, deletes all these entries afterwards.

The CommitLogCleanupJob, runs every 30 seconds and attempts to clean the old entries remaining from the profile_history_commit_log. It takes chunks of 100 profiles at a time, iterates over each profiles and reads all commit logs associated with it, and attempts to index them in elastic. If successful, the commit logs of that specific profile are deleted.

IDP Sessions

Modifications on IDP sessions are asynchronously saved in an in-memory buffer. The configurations for this buffer can be found in “/shibboleth/veridium-integration.json”, in the elasticIdpSessionsConfig field. The buffer is flushed whenever its size exceeds elasticBufferSize or if the last flush was more than elasticBufferFlushTimeoutMs miliseconds ago. To ensure the buffer is periodically flushed, an async worker checks every elasticBufferFlushWorkerPeriodMs miliseconds if the buffer should be flushed, and it flushes it if it’s the case. Also, every entry has a TTL: in case the ES indexing fails repeatedly and the entry has stayed more than elasticBufferRecordTtlMs miliseconds in the buffer, it will be removed from the buffer.

Action Logs

Action Logs are asynchronously saved whenever an action is performed. However, there is no temporary storage for action log data: if the ES indexing fails, it will not be retried.

Sessions

Session History entries are saved in the sessions_commit_log table, and the CommitLogCleanupJob is responsible for cleanup.