Photo by Brands&People on Unsplash

Elasticsearch is a great tool to gathering logs and metrics from various sources. It provides lots of default handlings for us so that the best user experience is cherished. However, there are cases that default handlings might not be optimal (especially in Production environment); hence today we would dive into ways to avoid “mappings pollution”.

what is a “mapping” and why is there a pollution?

Like any datastore solutions, there must be a schema equivalent setting to illustrate how a data field would be treated (e.g. stored and processed / analysed); this setting is known as “Mapping” under Elasticsearch.

Unlike the majority, Elasticsearch provides default handlings for schema settings if…


Welcome to the #1 post of the “golang gotchas” series. Today we will talk about the Singleton Pattern. This series is organised as follows:

  • code samples,
  • gotchas areas, and
  • theories

We know developers are very efficient in reading blogs and sort; hence wouldn’t it be nice if we can get all the code samples and gotchas in the first 10 minutes? Whilst the boring (but important) theories are left at the very end of the reading. Now we do have a choice, to know more through the theories or simply just get moving on your code :)

Code — how it works

The source code…


Photo by Christophe Dion on Unsplash

On the previous blog, we know that there is a special processor named as “pipeline” which acts like a function for other ingest-pipelines to invoke. Today we would look into some pipeline-processor techniques based on condition switching logics :)

setting a value based on a source field

Take an example, we have a field named “categoryValue”. If this value equals to “plant” then the field “categoryCode”’s value would be set to “A”. The following is the logic matrix:

  • categoryValue = “plant”, categoryCode = “A”
  • categoryValue = “animal”, categoryCode = “B”

The corresponding pipeline can be written as follows:

The switching logic is based on the “set” processor’s…


courtesy from http://www.freepik.com by almumtazza

Hello again~ Last time we have gone through how to deploy a 3-node elasticsearch cluster + 1 kibana instance under docker. In this tutorial, we would deploy a metricbeat to monitor the health and system metrics of the running containers.

PS. if you missed the previous tutorial, click here for more information https://quoeamaster.medium.com/deploying-elasticsearch-and-kibana-with-docker-86a4ac78d851

why need monitoring and why need metricbeat???

A commonly asked question but seldom got answered. First of all, no matter if we are deploying docker containers or the good old-fashion metal boxes (i.e. physical servers), it is crucial to monitor their health. The major concern is when we have several servers running in the…


In the good old days, deploying servers usually involve a set of Metal-Box(s), also they are allocated in a very “safe” place (usually data centre) to avoid all sorts of natural or human disturbances. This is the time when the infrastructure team handles all the hardware purchases and picking up the responsibility to look after the production machines. This is also the time when scaling up (scaling down is a very very rare use case) is a challenge as hardware capabilities hinder how much the production server(s) go.

Gone were the days~ The modern approach to deploy servers (or micro-services)…


Education photo created by wayhomestudio — www.freepik.com

Lately when I am looking around in the kibana UI, I found an interesting thing…


courtesy from rawpixel.com and http://www.freepik.com

Sometimes we do need to mask out confidential information before employing them for analysis. In this blog, we would take the challenge to use filebeat (instead of logstash or elasticsearch ingest node).

the filebeat way

First of all, we need to prepare a dataset. For simplicity, it would be like the following:

The dataset is in csv format, with the first field “client_id”, second field “transaction_date” and the third field “amount”. Assume we would like to mask out portions of the first field “client_id”.

If you have been using filebeat for sometime, you probably noticed that filebeat will treat each line of…


This is the 1st blog for the series — Modular Golang; hence would be focusing on some discussions about golang as a language and the challenges in developing apps with it. The coming blogs in this series would focus on use-cases and scenarios on how to design the app by a modular approach. Since app development is an on-going process (where new ideas and methodologies pop out nearly every day); the points discussed in this series might still be valid after a few months / years, however do bare in mind that outdated ideas and methodologies should be replaced when…


courtesy: from Photo by Littlehampton Bricks from Pexels

Foreword…

If you ever work with Elasticsearch before, you know that a specific Query syntax (named QueryDSL) would be employed to instruct how Elasticsearch works for us. QueryDSL is a human readable format expressed in JSON, typically it is quite straightforward and easy to write out a query to handle search requests; however… if we are supposed to handle dynamic query conditions… then we would very soon fall into a mud of miserables where we need to take care of lots of String level modifications and make sure the query works.

PS. We are going to build the solution on top…


photo from https://www.pexels.com/photo/abstract-art-blur-bright-373543/

Foreword…

When we are developing mobile phone apps, it is not uncommon to make API calls through the http protocol. For some cases, the target API provider would provide a library for some specific programming languages which would ease the development at once. However for most situations, the API provider only exposes the REST API layer and let developers build its own http connectivity code.

In this blog, we would take a look at the http package on Flutter, and how we could make use of the package to build our own http connectivity code.

The Sample app

We build on top of a…

devops terminal

a java / golang / flutter developer, a big data scientist, a father :)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store