Photo by Christophe Dion on Unsplash

On the previous blog, we know that there is a special processor named as “pipeline” which acts like a function for other ingest-pipelines to invoke. Today we would look into some pipeline-processor techniques based on condition switching logics :)

setting a value based on a source field

Take an example, we have a field named “categoryValue”. If this value equals to “plant” then the field “categoryCode”’s value would be set to “A”. The following is the logic matrix:

The corresponding pipeline can be written as follows:

The switching logic is based on the “set” processor’s…


courtesy from http://www.freepik.com by almumtazza

Hello again~ Last time we have gone through how to deploy a 3-node elasticsearch cluster + 1 kibana instance under docker. In this tutorial, we would deploy a metricbeat to monitor the health and system metrics of the running containers.

PS. if you missed the previous tutorial, click here for more information https://quoeamaster.medium.com/deploying-elasticsearch-and-kibana-with-docker-86a4ac78d851

why need monitoring and why need metricbeat???

A commonly asked question but seldom got answered. First of all, no matter if we are deploying docker containers or the good old-fashion metal boxes (i.e. physical servers), it is crucial to monitor their health. The major concern is when we have several servers running in the…


In the good old days, deploying servers usually involve a set of Metal-Box(s), also they are allocated in a very “safe” place (usually data centre) to avoid all sorts of natural or human disturbances. This is the time when the infrastructure team handles all the hardware purchases and picking up the responsibility to look after the production machines. This is also the time when scaling up (scaling down is a very very rare use case) is a challenge as hardware capabilities hinder how much the production server(s) go.

Gone were the days~ The modern approach to deploy servers (or micro-services)…


Education photo created by wayhomestudio — www.freepik.com

Lately when I am looking around in the kibana UI, I found an interesting thing…


courtesy from rawpixel.com and http://www.freepik.com

Sometimes we do need to mask out confidential information before employing them for analysis. In this blog, we would take the challenge to use filebeat (instead of logstash or elasticsearch ingest node).

the filebeat way

First of all, we need to prepare a dataset. For simplicity, it would be like the following:

The dataset is in csv format, with the first field “client_id”, second field “transaction_date” and the third field “amount”. Assume we would like to mask out portions of the first field “client_id”.

If you have been using filebeat for sometime, you probably noticed that filebeat will treat each line of…


This is the 1st blog for the series — Modular Golang; hence would be focusing on some discussions about golang as a language and the challenges in developing apps with it. The coming blogs in this series would focus on use-cases and scenarios on how to design the app by a modular approach. Since app development is an on-going process (where new ideas and methodologies pop out nearly every day); the points discussed in this series might still be valid after a few months / years, however do bare in mind that outdated ideas and methodologies should be replaced when…


courtesy: from Photo by Littlehampton Bricks from Pexels

Foreword…

If you ever work with Elasticsearch before, you know that a specific Query syntax (named QueryDSL) would be employed to instruct how Elasticsearch works for us. QueryDSL is a human readable format expressed in JSON, typically it is quite straightforward and easy to write out a query to handle search requests; however… if we are supposed to handle dynamic query conditions… then we would very soon fall into a mud of miserables where we need to take care of lots of String level modifications and make sure the query works.

PS. We are going to build the solution on top…


photo from https://www.pexels.com/photo/abstract-art-blur-bright-373543/

Foreword…

When we are developing mobile phone apps, it is not uncommon to make API calls through the http protocol. For some cases, the target API provider would provide a library for some specific programming languages which would ease the development at once. However for most situations, the API provider only exposes the REST API layer and let developers build its own http connectivity code.

In this blog, we would take a look at the http package on Flutter, and how we could make use of the package to build our own http connectivity code.

The Sample app

We build on top of a…


Sometimes we might came across signs and documents which are informative and the next action we do is to take out our phone to take a photo. However wouldn’t it be nice if the photo’s contents could be recognised and converted into text? Thanks to the advance in text-recognition technologies, OCR related operations could be done much much much easier than ever — we don’t need a setup a server to run the recognition algorithms anymore… whew~

This blog would be focusing on how to add in text-recognition capability into a Flutter mobile phone app — yep~ since it is…


screenshot on the “transform” UI — Kibana

Video Tutorial at here: https://www.youtube.com/watch?v=WwPuWYNelpY

Elasticsearch is a great tool for ingesting data and perform fast aggregations (also known as analytics). All data logs ingested are treated as individual events in which theoretically have no dependance on other entries. However logically this might not be the case.

In a eCommerce platform, when a user checks out the shopping cart and paid, a corresponding event (or a set of related application logs) would be ingested by Elasticsearch. So this entry(s) is treated as an individual event which sounds reasonable; however, this particular user might have purchased on this eCommerce platform for…

devops terminal

a java / golang / flutter developer, a big data scientist, a father :)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store