Shipping Docker logs from EC2 to Elastic Cloud

EC2 container comes with a pre-installed filebeat and metricbeat, however, the filebeat/metricbeat version might not be the same as provided by Elastic cloud. Elastic cloud has the most recent filebeat version. In the example, we will use filebeat version 6.8, which I had as I have not upgraded AWS Elasticsearch for a while.

Login to EC2 container where your docker application is running. Type the following command to look for the path of filebeat

Now we that we know that filebeat is located in the path /etc/filebeat/*

Update your login permission with more administrative power by running sudo su to make sure you have sufficient permission to update files. change directory to point at filebeat directory and list the content

Open filebeat.yml using your preferred editor (nano, vi, etc), Edit the file, and replace filebeat.prospertor with the use docker input to read logs from Docker containers.

This input searches for container logs under its path, and parse them into common message lines, extracting timestamps too. To make sure we are able to read from all the container we will use containers.ids: ‘*’

if want to configure for the specific container, then we can look for a specific container id or list of ids under the following path /var/lib/docker/containers

If you are getting permission error to this path, make sure you are using administrative users.

save the filebeat.yml file after applying the above changes. Now its time to check configurations are correct. Let's restart filebeat

To restart the filebeat you can run either of these commands

service filebeat stop followed by service filebeat start or you can club this in one single operation as service filebeat restart

Now, let's go to the Elastic cloud console, and open Index management to validate if we are getting filebeat-*

The next step we need to perform would be to create an index pattern, for some of you, it might have been preconfigured, in my case as my initial use-case was only for APM, I have to create the index pattern

To create an index pattern, click on index pattern and create a new index pattern as shown in the screenshot=

follow the steps and save the pattern

On complete, you will notice some new entries

We are all set