Elastic Search entities index procedure
Devices
Any modification on the device entity generates a commit log entry in the device_history_commit_log table. Right after this save, a new thread attempts to asynchronously index all the commit logs of that specific device, and, if successful, deletes all these entries afterwards.
The CommitLogCleanupJob, runs every 30 seconds and attempts to clean the old entries remaining from the device_history_commit_log. It takes chunks of 100 devices at a time, iterates over each device and reads all commit logs associated with it, and attempts to index them in elastic. If successful, the commit logs of that specific device are deleted.
Accounts
Any modification on the device entity generates a commit log entry in the account_history_commit_log table. Right after this save, a new thread attempts to asynchronously index all the commit logs of that specific account, and, if successful, deletes all these entries afterwards.
The CommitLogCleanupJob, runs every 30 seconds and attempts to clean the old entries remaining from the account_history_commit_log. It takes chunks of 100 accounts at a time, iterates over each account and reads all commit logs associated with it, and attempts to index them in elastic. If successful, the commit logs of that specific account are deleted.
Profiles
Any modification on the profile entity generates a commit log entry in the profile_history_commit_log table. Right after this save, a new thread attempts to asynchronously index all the commit logs of that specific profile, and, if successful, deletes all these entries afterwards.
The CommitLogCleanupJob, runs every 30 seconds and attempts to clean the old entries remaining from the profile_history_commit_log. It takes chunks of 100 profiles at a time, iterates over each profiles and reads all commit logs associated with it, and attempts to index them in elastic. If successful, the commit logs of that specific profile are deleted.
IDP Sessions
Modifications on IDP sessions are asynchronously saved in an in-memory buffer. The configurations for this buffer can be found in “/shibboleth/veridium-integration.json”, in the elasticIdpSessionsConfig
field. The buffer is flushed whenever its size exceeds elasticBufferSize
or if the last flush was more than elasticBufferFlushTimeoutMs
miliseconds ago. To ensure the buffer is periodically flushed, an async worker checks every elasticBufferFlushWorkerPeriodMs
miliseconds if the buffer should be flushed, and it flushes it if it’s the case. Also, every entry has a TTL: in case the ES indexing fails repeatedly and the entry has stayed more than elasticBufferRecordTtlMs
miliseconds in the buffer, it will be removed from the buffer.
Action Logs
Action Logs are asynchronously saved whenever an action is performed. However, there is no temporary storage for action log data: if the ES indexing fails, it will not be retried.
Sessions
Session History entries are saved in the sessions_commit_log table, and the CommitLogCleanupJob is responsible for cleanup.