Elasticsearch is a great tool to gathering logs and metrics from various sources. It provides lots of default handlings for us so that the best user experience is cherished. However, there are cases that default handlings might not be optimal (especially in Production environment); hence today we would dive into ways to avoid “mappings pollution”.

what is a “mapping” and why is there a pollution?

Like any datastore solutions, there must be a schema equivalent setting to illustrate how a data field would be treated (e.g. stored and processed / analysed); this setting is known as “Mapping” under Elasticsearch.

Unlike the majority, Elasticsearch provides default handlings for schema settings if…

Welcome to the #1 post of the “golang gotchas” series. Today we will talk about the Singleton Pattern. This series is organised as follows:

• code samples,
• gotchas areas, and
• theories

We know developers are very efficient in reading blogs and sort; hence wouldn’t it be nice if we can get all the code samples and gotchas in the first 10 minutes? Whilst the boring (but important) theories are left at the very end of the reading. Now we do have a choice, to know more through the theories or simply just get moving on your code :)

Code — how it works

The source code…

On the previous blog, we know that there is a special processor named as “pipeline” which acts like a function for other ingest-pipelines to invoke. Today we would look into some pipeline-processor techniques based on condition switching logics :)

setting a value based on a source field

Take an example, we have a field named “categoryValue”. If this value equals to “plant” then the field “categoryCode”’s value would be set to “A”. The following is the logic matrix:

• categoryValue = “plant”, categoryCode = “A”
• categoryValue = “animal”, categoryCode = “B”

The corresponding pipeline can be written as follows:

The switching logic is based on the “set” processor’s…

Hello again~ Last time we have gone through how to deploy a 3-node elasticsearch cluster + 1 kibana instance under docker. In this tutorial, we would deploy a metricbeat to monitor the health and system metrics of the running containers.

PS. if you missed the previous tutorial, click here for more information https://quoeamaster.medium.com/deploying-elasticsearch-and-kibana-with-docker-86a4ac78d851

why need monitoring and why need metricbeat???

A commonly asked question but seldom got answered. First of all, no matter if we are deploying docker containers or the good old-fashion metal boxes (i.e. physical servers), it is crucial to monitor their health. The major concern is when we have several servers running in the…

In the good old days, deploying servers usually involve a set of Metal-Box(s), also they are allocated in a very “safe” place (usually data centre) to avoid all sorts of natural or human disturbances. This is the time when the infrastructure team handles all the hardware purchases and picking up the responsibility to look after the production machines. This is also the time when scaling up (scaling down is a very very rare use case) is a challenge as hardware capabilities hinder how much the production server(s) go.

Gone were the days~ The modern approach to deploy servers (or micro-services)…

Lately when I am looking around in the kibana UI, I found an interesting thing…

Sometimes we do need to mask out confidential information before employing them for analysis. In this blog, we would take the challenge to use filebeat (instead of logstash or elasticsearch ingest node).

the filebeat way

First of all, we need to prepare a dataset. For simplicity, it would be like the following:

The dataset is in csv format, with the first field “client_id”, second field “transaction_date” and the third field “amount”. Assume we would like to mask out portions of the first field “client_id”.

If you have been using filebeat for sometime, you probably noticed that filebeat will treat each line of…

This is the 1st blog for the series — Modular Golang; hence would be focusing on some discussions about golang as a language and the challenges in developing apps with it. The coming blogs in this series would focus on use-cases and scenarios on how to design the app by a modular approach. Since app development is an on-going process (where new ideas and methodologies pop out nearly every day); the points discussed in this series might still be valid after a few months / years, however do bare in mind that outdated ideas and methodologies should be replaced when…