Deploying Metricbeat side by side with Elasticsearch in docker
Hello again~ Last time we have gone through how to deploy a 3-node elasticsearch cluster + 1 kibana instance under docker. In this tutorial, we would deploy a metricbeat to monitor the health and system metrics of the running containers.
PS. if you missed the previous tutorial, click here for more information https://quoeamaster.medium.com/deploying-elasticsearch-and-kibana-with-docker-86a4ac78d851
why need monitoring and why need metricbeat???
A commonly asked question but seldom got answered. First of all, no matter if we are deploying docker containers or the good old-fashion metal boxes (i.e. physical servers), it is crucial to monitor their health. The major concern is when we have several servers running in the field, how efficient is it to identify which server is having an issue? Or maybe how easy is it to determine when a particular server is going to crash? The answer could be found within the monitoring data (if we ever collect them at the beginning).
Based on experiences and team culture, administrators usually write their own scripts or programs to collect server metrics. Indeed this is a fairly straight-forward approach and usually works well… until many server instances are deployed. Are the scripts and programs scalable to handle multi-server monitoring? How many customisations and how easy is it to monitor different systems (e.g. OS, databases, application servers and more). If the answers for the above are not firm, we probably need a better solution.
Metricbeat from elastic (the company) provides a single executable which help to ingest the most common servers / services metrics including host (i.e. the OS level), databases (mySQL, postgreSQL), message-queues (kafka, rabbitMQ) and more. Different from home brew solutions, metricbeat is maintained and actively under development to support metric consumptions; now all we need to do is to maintain configuration files instead of the logics in our scripts / programs.
add a metricbeat service
First of all, download the latest version of metricbeat from docker:
docker pull docker.elastic.co/beats/metricbeat:7.11.1
Now modify the docker-compose.yml as follows:
We added a service named “metricbeat” and declared the target elasticsearch host through an environment variable (ELASTICSEARCH_HOSTS). Also a new volume (for data storage) is added for this service (metricbeat-data01). The network for this service should be set as “es-network”, the reason is the service should be on the same network as our target elasticsearch host / node.
Let’s kick start the services by issuing:
docker-compose up
Within a minute or two, the 3 services (elasticsearch node, kibana and metricbeat) should be online. To verify the results, open kibana at http://localhost:5601 and navigate to the metrics app
Congratulations we have successfully added metricbeat monitoring on our cluster! The data collected could be shown on the metrics app which is reflecting the OS / host metrics including cpu, memory, loading, network, processes and more.
Displaying OS / host metrics is quite informative, however sometimes we probably want to know how a server / service is behaving as well. Take an example, could we monitor our elasticsearch node’s health? If you click on the “stack monitoring” app by now, this page would be shown:
Simply means… no elasticsearch monitoring data available :(
understanding metricbeat modules
For the modern versions of the elastic stack, monitoring of elasticsearch nodes suggests to enable a metricbeat module named “elasticsearch-xpack”. So what are metricbeat modules??? A simple sentence — modules contains the logic to talk with the targeted server / service and maintains sets of logical groupings of metrics for collection; hence it is like a configuration manager.
Modules are declared in a yml file and stored under the {metricbeat_root}/modules.d folder. An example of the elasticsearch-xpack.yml as follows:
metricbeat.modules:
- module: elasticsearch
metricsets:
— node
— node_stats
#- index
#- index_recovery
#- index_summary
#- shard
#- ml_job
period: 10s
hosts: [“http://localhost:9200”]#username: “elastic”
#password: “changeme”
#ssl.certificate_authorities: [“/etc/pki/root/ca.pem”]#index_recovery.active_only: true
xpack.enabled: false
#scope: node
For most cases, we would modify which sets of metrics to collect (i.e. metricsets setting), hosts for monitoring (e.g. localhost:9200 or node01:9200) and activate the xpack monitoring feature (i.e. xpack.enabled).
Now the challenge ahead of us is … how to update the elasticsearch-xpack.yml file??? Well… one way is we start up the services and then telnet / ssh into the metricbeat container and update the config file. This way is probably the easiest but but but it is not scalable at all and introducing manual operations which is hard to maintain eventually. A less dirty solution definitely pays.
build a customised metricbeat docker image
Like what we have discussed earlier, we are trying to maintain only configuration files instead of script / program logics (also manual operations). Hence a cleaner solution for the above challenge is to build a customised docker image with the updated metricbeat.yml.
Create the metricbeat.yml as follows:
We declared the “elasticsearch” module to be enabled and the target host is node01:9200. Since the “hosts” configuration is an array, it is possible to setup only 1 metricbeat instance and consume multi-nodes’ metric.
Next, let’s update the docker-compose.yml as follows:
The only change of the configuration file is to replace the “image” config with a “build” config. “build” simply means we are building a custom image from a local Dockerfile config. Dockerfile contains the steps to build a customised image and our version as follows:
It is fairly simple, we set the base docker image as docker.elastic.co/beats/metricbeat:7.11.1; then we copy the customised metricbeat.yml to the image’s /usr/share/metricbeat path. Finally use root user to change the access right for the metricbeat.yml and done.
Next is to build the image but issuing:
docker-compose build
If all is good, our image would be created in no time.
Time to spin up the services and verify the results:
docker-compose up
Navigate to kibana > stack monitoring app. This time we should see a different page.
Hooray~ We did it, metricbeat is collecting elasticsearch node’s metrics.
This approach works like a charm but but but what if we want to avoid building or updating the metricbeat docker image on every deployment change? There is another solution named as hint-based configurations.
what do you mean by … hint-based???
Metricbeat has added a super cool feature named as hint-based configuration and the idea is simple, our target server / service would be started with a set of hints (in actual, docker labels); when metricbeat service is activated, it would try to sniff the network for such hints. Assume hints are provided and accurately configured, now metricbeat could start collecting metrics immediately.
This solution removes the needs to build a new docker image on cluster changes, it also removes the need to hard code metricbeat.yml configurations to some degrees as well.
The first step for hint-based configuration to work is to build a custom metricbeat image. Yes, we still need to build a custom image for the LAST time (promised). The following is the metricbeat.yml for the image:
Noted that a section named as “metricbeat.autodiscover” is added and declared our auto-discover functionality is based on a docker environment. That’s it, pretty simple isn’t it? Next is update the docker-compose.yml:
We have added a “labels” section under the node01 service. These label entries declared which metricbeat module would be supported, in our case the “elasticsearch” monitoring module; plus which metricset(s), in our case the “node” and “node_stats” metricsets. Now it is time to build the custom image by issuing:
docker-compose build
To startup all services, issuing the following:
docker-compose up
Within a minute of two, all services (elasticsearch node01, kibana and metricbeat) should be online. Navigate to kibana > stack monitoring:
PS. this time, we will pick the “self monitoring” feature instead. At the time of writing, there seems to have no way to pass a configuration named “xpack.enabled = true” through label based hints; hence the only valid option left is the “self monitoring”
Congratulations~ We just created a hint based metricbeat configuration. Starting from now on, if we need to add / remove any elasticsearch nodes, no changes should be applied to the metricbeat docker image anymore; instead the corresponding node’s configuration would need to expose whatever metrics module it wants to expose to metricbeat~
As discussed earlier this approach ease maintenance efforts by keeping config files updated instead of re-building docker images.
TLDR hints, tips and traps
Similar to the last tutorial, I would like to share some hints, tips and traps met.
Q. Um… what is the difference between a Dockerfile and docker-compose.yml?
Dockerfile is for building a new docker image; the contents of the file contains instructions to build that image.
The docker-compose.yml is for service deployment, and each service is dependent on a docker image. So NEVER mix them up.
Q. Hey, I see that a new volume (/var/run/docker.sock:/var/run/docker.sock) is added in the docker-compose.yml, what’s that for?
For the hint-based metricbeat service to work, we would need the support from docker to pass out hints (in actual, docker image labels); hence we would need to access the docker service located under the socker /var/run/docker.sock
Q. Hm… there is also a new entry named “user: root” ???
Yes, for metricbeat to run smoothly, we might need to use the root user so that metricbeat could access system function calls to gather specific service / server metrics. In our case, root user is required to access the docker socket.
A hint on whether a root user is required is when the console log shows errors on access right issues.
closings
Nice journey again~ Till now we have gone through the following:
- why monitoring is important
- why should we try metricbeat instead of home brew scripts and programs
- creating our first metricbeat service under docker-compose.yml — collecting OS / host level metrics
- understanding what is metricbeat modules — the key to monitor server / service metrics
- creating our second metricbeat service with a customised docker image — activating “elasticsearch” monitoring module
- understanding what is hint-based monitoring approach and why we should consider adopting — an image build-free solution to adopt changes in your cluster / deployments
- creating our third metricbeat service with hint-based capability — collecting “elasticsearch” monitoring metrics
For the next journey, we would look at how kubernetes helps to manage our 3 node cluster deployment, also discovering the side-car approach to add back metricbeat for health monitoring. Stay tune~ :)
