Data structures

Keyspace concept

All Elastic resources managed by Veridium application (data indices, templates and lifecycle policies) are created under a “keyspace“ concept. This is done by having a prefix in each resource name (ex: 'veridium.'), because there is no support for a keyspace in Elasticsearch.

The keyspace is convenient to have a better isolation, multi-tenant capabilities, improved management.

The keyspace is defined in /elasticsearch.json configuration node in Zookeeper and by default is the one from Cassandra.

JSON

{
    "clientConnection": {
        "hosts": "10.204.0.107",
        "port": 9095,
        "keyspace": "veridium",
        "apiKey": "S1NhdzRZOEJwQVhEOHJpT3ZIYko6ODRiY1hnNmhSMzZlTnJDWlNVT25Tdw==",
        "credentials": {
            "user": "elastic",
            "passwd": "ChangeMe"
        },
        "sslOptions": {
            "enabled": true,
            "sslVerificationMode": "certificate"
        }
    }
}

The Veridium Es client integration uses the keyspace for every request made to the Elasticsearch cluster and this way limits the visibility over the resources from configured keyspace only.

Templates

Each Veridium index is managed by using Elastic templates capabilities:

component templates - one or multiple component templates to define custom data mapping types. By default the type mapping is done dynamically by the Elastic. The component templates specify custom mappings or disable certain fields to be indexed (for space and efficiency concerns).
index template - automatically applies settings, aliases and. mappings to the indices. It composes one or several component templates for data type mappings, set the replication factor, number of shards and more.
Lifecycle policies - define lifecycle phases for each index (one template for each index)

All these templates are stored in configurations as well (Zookeeper) and updated on each version installation.

Config resources:

/elasticsearch/index-templates.json
/elasticsearch/component-templates.json
/elasticsearch/lifecycle-policies.json

Currently available entities in Elastic search