exploring the new index_template in Elasticsearch 7.8
Lately when I am looking around in the kibana UI, I found an interesting thing…
So there are 2 types of index templates(??). To be honest, when I click around I could not tell the difference. Therefore I would go into the official documentations instead https://www.elastic.co/guide/en/elasticsearch/reference/7.10/index-templates.html
a story of TWO tales
after some readings, things become much much clearer. The new index template(s) is the enhanced version which heavily focus on re-usable components.
The truth is when we declare index templates in the old days, we might need to copy-and-paste some common settings and mappings to every targeted template. Of course, there are workarounds by applying the “order” setting within each template, so that setting / mapping “inheritance” could be achieved without the error prone copy-and-paste process. However… depending on the “order” setting itself could be complicated and frustrating. Definitely a better solution needs to be evolved.
At this point, make it clear that the old way to create index templates are now known as “legacy index templates”. Whilst the component based approach is taking the name “index templates”.
component based solution
In order to reduce the complication of template inheritance issue; a component based solution was introduced since version 7.8. Make it simple, the re-usable parts (no matter it is the settings or mappings) could be extracted as a component. Then, to architect the targeted index template, all we need to do is to declare which re-usable components would be employed. Let’s take an example as follows:
we first declare a common set of mappings and settings and persist under the _component_template endpoint / api. Not surprising, the component template syntax is nearly identical to the good old index_template’s.
A set of rules are declared in our components:
- any string data prefixed “s_” would be converted to a keyword,
- any long data prefixed “i_” would be converted to an integer,
- declared a date field — “@timestamp”,
- number of primary shards would be 3 and number of replicas to be 1
Next step is to architect the targeted index template; assume we are building an eCommerce platform and a shopping cart index is required, we also know that most of the fields involved would follow a naming convention; hence this is a good example for re-usable components!
To declare the index template with components, try this:
everything seems just the same as usual except the new setting — “composed_of” which is declaring which re-usable component(s) would be included as building blocks of this template. Clearly we would reuse the “core” and “core_settings” components. To validate whether the template works, simply ingest a new document:
let’s check the results:
- field “i_quantity” is an integer instead of long,
- field “s_discount_code” is a keyword instead of text + keyword,
- “@timestamp” is also available within the mappings, even though we do not ingest this field in the example above,
- for the other fields that do NOT fit with the rules such as sales_person.id and sales_person.name; both fields get back the default text + keyword combination,
- number of primary shards is 3 and replica counts is 1
cool~ the shopping_cart works as expected~ Plus… we never need to worry about the “order” of inheritance anymore! Remember, if we would need a template to inherit a couple of re-usable components, simply include them in the template definition under the “composed_of” configuration.
Closing
Till now, we have explored a few things:
- the difference between the legacy and new index templates
- the rationale and approach to component based templates
- an example on how to architect a targeted index template built from re-usable components
PS. do remember that this feature is available at 7.8 or newer versions. Also, creating templates in the legacy way is still supported for the moment.
