How to Install Elasticsearch, Fluentd and Kibana (EFK) Logging Stack on Ubuntu 22.04
Log monitoring and analysis is an essential part of server or container infrastructure and is useful when handling complex applications. One of the popular logging solutions is the Elasticsearch, Fluentd, and Kibana (EFK) stack. Before going further into the tutorial, let us learn about the stack's components.
Elasticsearch is a real-time, distributed, and scalable search engine that allows for full-text search and analytics. It is used to index and search through large amounts of data. It is commonly deployed alongside Kibana, a powerful data visualization dashboard for Elasticsearch. Kibana allows you to explore the Elasticsearch log data and build dashboards and queries to gain insight into your application. Fluentd collects, transforms, and ships the log data to the Elasticsearch backend.
In this tutorial, we will install the EFK stack using Docker on a Ubuntu 22.04 machine and send container logs to Kibana after filtering and transforming them using Fluentd.
Prerequisites
-
A server running Ubuntu 22.04 with a minimum of 6GB of RAM.
-
A non-root user with sudo privileges.
-
The uncomplicated Firewall(UFW) is enabled and running.
-
A Fully Qualified domain name (FQDN) pointing to the server like,
kibana.example.com. -
Everything is updated.
$ sudo apt update && sudo apt upgrade
Step 1 - Configure Firewall
Before installing any packages, the first step is configuring the firewall to allow HTTP and HTTPS connections.
Check the status of the firewall.
$ sudo ufw status
You should see something like the following.
Status: active
To Action From
-- ------ ----
OpenSSH ALLOW Anywhere
OpenSSH (v6) ALLOW Anywhere (v6)
Allow HTTP and HTTPs ports.
$ sudo ufw allow http
$ sudo ufw allow https
Check the status again to confirm.
$ sudo ufw status
Status: active
To Action From
-- ------ ----
OpenSSH ALLOW Anywhere
80/tcp ALLOW Anywhere
443 ALLOW Anywhere
OpenSSH (v6) ALLOW Anywhere (v6)
80/tcp (v6) ALLOW Anywhere (v6)
443 (v6) ALLOW Anywhere (v6)
Step 2 - Install Docker and Docker Compose
Add Docker's official GPG key.
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker.gpg
Run the following command to add the Docker repository.
$ echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Update the system to include Docker's repository.
$ sudo apt update
Install Docker and the Docker compose plugin.
$ sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin
This tutorial will be using the Docker Compose v2 plugin instead of the older legacy binary. Therefore, the command for running it has changed from docker-compose to docker composeand this is reflected here.
Docker runs with elevated privileges so you will need to use sudo frequently to run commands. The better option is to add your Linux user account to the docker user group.
$ sudo usermod -aG docker ${USER}
The ${USER} variable picks up the currently logged-in system account. If you are not logged in with the user you want to give privileges to, replace ${USER} with the username.
To apply for the new group membership, log out of the server and back in, or use the following command. You will be prompted for the user's password.
$ su - ${USER}
Step 3 - Create Docker Compose File
First, create the directory for the EFK project.
$ mkdir ~/efk
Switch to the directory.
$ cd ~/efk
Create and open the docker-compose.yml file for editing.
$ nano docker-compose.yml
Paste the following code in it.
services:
# Deploy using the custom image automatically be created during the build process.
fluentd:
build: ./fluentd
volumes:
- ./fluentd/conf:/fluentd/etc
links: # Sends incoming logs to the elasticsearch container.
- elasticsearch
depends_on:
- elasticsearch
ports: # Exposes the port 24224 on both TCP and UDP protocol for log aggregation
- 24224:24224
- 24224:24224/udp
elasticsearch:
image: elasticsearch:8.7.1
expose:
- 9200
environment:
- discovery.type=single-node # Runs as a single-node
- xpack.security.enabled=false
volumes: # Stores elasticsearch data locally on the esdata Docker volume
- esdata:/usr/share/elasticsearch/data
kibana:
image: kibana:8.7.1
links: # Links kibana service to the elasticsearch container
- elasticsearch
depends_on:
- elasticsearch
ports:
- 5601:5601
environment: # Defined host configuration
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
# Define the Docker volume named esdata for the Elasticsearch container.
volumes:
esdata:
Save the file by pressing Ctrl + X and entering Y when prompted. We have configured to start three services, one for each, Fluentd, Elasticsearch, and Kibana.
For Fluentd, we will build a container instead of a readymade image. The build files for Fluentd will be set up in the next step. We have mounted a directory for its build files, and a volume for the configuration files, and exposed port 24224 on both TCP and UDP protocol for log aggregation.
The next service is Elasticsearch and we are using the latest version available at the time of writing this tutorial. We have exposed it via port 9200 and set up a few environment variables so that we can run it as a single-node cluster and have disabled the security features. This is not recommended usually but enabling security is out of the scope of this tutorial. We have also mounted a local volume for the Elasticsearch data.
Finally, we configure Kibana and expose it via port 5601 which will be used to access the dashboard. We also set up a variable to configure the Elasticsearch host for it to access.
Step 4 - Set up Fluentd Build Files
Create Fluentd and the configuration directory.
$ mkdir fluentd/conf -p
Run the tree command to verify the directory structure.
$ tree
It should look like the following.
Switch to the Fluentd directory.
$ cd fluentd
Create and open the Dockerfile for editing.
$ nano Dockerfile
Paste the following code in it. This code pulls the Fluentd Debian Docker image and installs the Fluentd plugin for Elasticsearch.
# fluentd/Dockerfile
FROM fluent/fluentd:v1.16-debian-1
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.3.0"]
USER fluent
Save the file by pressing Ctrl + X and entering Y when prompted.
Switch to the configuration directory.
$ cd conf
Create and open the fluentd.conf file for editing.
$ nano fluentd.conf
Paste the following code in it.
# bind fluentd on IP 0.0.0.0
# port 24224
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
# sendlog to the elasticsearch
# the host must match to the elasticsearch
# container service
<match *.**>
@type copy
<store>
@type elasticsearch_dynamic
hosts elasticsearch:9200
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
tag_key @log_name
include_timestamp true
flush_interval 30s
</store>
<store>
@type stdout
</store>
</match>
Save the file by pressing Ctrl + X and entering Y when prompted.
The source directive above uses the forward plugin which turns Fluentd into a TCP endpoint to accept TCP packets.
The match directive looks for events with matching tags which in this case means it matches all the events. We will use the elasticsearch_dynamic plugin for storage which allows configuration values to be specified dynamically. The hosts field specifies the hostname for the Elasticsearch application which is the service name in the Docker compose file. The logstash_format is set to true which means Fluentd uses the conventional name format logstash-%Y.%m.%dlogstash-%Y.%m.%d. The prefix name to write the events is set to fluend. The include_tag_key is set to true which adds the Fluentd tag in the JSON format. The tag_key is the field name to extract for the tag. Setting the variable include_timestamp to true adds a timestamp field to the log. The flush_interval specifies the interval between data flushes. We also use the stdout plugin to print events/logs to the standard output.
Step 5 - Run the Docker Containers
Switch back to the EFK directory.
$ cd ~/efk
Start the containers using the following command.
$ docker compose up -d
Check the status of the running containers.
$ docker ps
b3780c311154 efk-fluentd "tini -- /bin/entryp…" 9 seconds ago Up 8 seconds 5140/tcp, 0.0.0.0:24224->24224/tcp, 0.0.0.0:24224->24224/udp, :::24224->24224/tcp, :::24224->24224/udp efk-fluentd-1
5a48f0a9ade1 kibana:8.7.1 "/bin/tini -- /usr/l…" 9 seconds ago Up 7 seconds 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp efk-kibana-1
dab3a0ab0312 elasticsearch:8.7.1 "/bin/tini -- /usr/l…" 9 seconds ago Up 8 seconds 9200/tcp, 9300/tcp efk-elasticsearch-1
You can also use the following command for it.
$ docker compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
efk-elasticsearch-1 elasticsearch:8.7.1 "/bin/tini -- /usr/l…" elasticsearch 37 seconds ago Up 36 seconds 9200/tcp, 9300/tcp
efk-fluentd-1 efk-fluentd "tini -- /bin/entryp…" fluentd 37 seconds ago Up 36 seconds 5140/tcp, 0.0.0.0:24224->24224/tcp, 0.0.0.0:24224->24224/udp, :::24224->24224/tcp, :::24224->24224/udp
efk-kibana-1 kibana:8.7.1 "/bin/tini -- /usr/l…" kibana 37 seconds ago Up 36 seconds 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp
Run the following commands to check the logs of the EFK build process.
$ docker logs efk-fluentd-1
$ docker logs efk-kibana-1
$ docker logs efk-elasticsearch-1
Inspect the Elasticsearch container. It will print out the detailed settings of the container.
$ docker inspect efk-elasticsearch-1
You can notice the IP address given to the container.
[
{
"Id": "dab3a0ab03120d3a7192045e1ea84fdd0f8fdb7819cc6d6780e05109d61e0b66",
"Created": "2023-05-04T09:58:00.256169904Z",
"Path": "/bin/tini",
"Args": [
"--",
"/usr/local/bin/docker-entrypoint.sh",
"eswrapper"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23619,
"ExitCode": 0,
"Error": "",
"StartedAt": "2023-05-04T09:58:00.563700803Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:59075530be34d3a06866f894ae9735f6d739a7a751ad45efb86dec3c9bd16836",
"ResolvConfPath": "/var/lib/docker/containers/dab3a0ab03120d3a7192045e1ea84fdd0f8fdb7819cc6d6780e05109d61e0b66/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/dab3a0ab03120d3a7192045e1ea84fdd0f8fdb7819cc6d6780e05109d61e0b66/hostname",
"HostsPath": "/var/lib/docker/containers/dab3a0ab03120d3a7192045e1ea84fdd0f8fdb7819cc6d6780e05109d61e0b66/hosts",
"LogPath": "/var/lib/docker/containers/dab3a0ab03120d3a7192045e1ea84fdd0f8fdb7819cc6d6780e05109d61e0b66/dab3a0ab03120d3a7192045e1ea84fdd0f8fdb7819cc6d6780e05109d61e0b66-json.log",
"Name": "/efk-elasticsearch-1",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "efk_default",
"PortBindings": {},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": [],
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"Mounts": [
{
"Type": "volume",
"Source": "efk_esdata",
"Target": "/usr/share/elasticsearch/data",
"VolumeOptions": {}
}
],
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/ee03648cf34e03601848b1769569b4d3bb7192db118102ca050215ba87060bbf-init/diff:/var/lib/docker/overlay2/51d6cfcb59e473a3f163e68984a1ba1325a2c816ed7925c4dffdefcf2e104d11/diff:/var/lib/docker/overlay2/b9c096454bda31f1cb2ea33f108be8b29b2e94827ebe94cc17563eb596b7cab1/diff:/var/lib/docker/overlay2/effe604c5b015ba02cf3b7a238bd3ff5dad7970a72e689ef5275fcf03fd0bcd1/diff:/var/lib/docker/overlay2/72fbf23251467ea2f6af8d9458c7fdd8fa3ef716eeafd9319ceff59d07d96788/diff:/var/lib/docker/overlay2/02094ec9e4ebb04371f782744a3a46852a00bf6fd7e8820d466a3576aeb9d5fc/diff:/var/lib/docker/overlay2/ce364cdd636b67e10c879aa152360d965d08fe456663ed8fbe78c3bd37bde6c7/diff:/var/lib/docker/overlay2/33bf44b475ea5ea249845b7eed75ded47dd9dc7877b9231fa4195b4753071945/diff:/var/lib/docker/overlay2/4f19bd8089599ef879075012c710ec464d8e0446fc0a0813850657dddd23a5dc/diff:/var/lib/docker/overlay2/a39a61b12d8565c6d5b33c17a04d47c8bd47609a787e0548fbac0d47d00eecc8/diff:/var/lib/docker/overlay2/cbd9d77eb9ed6b600511f9a676aab511d2aa2b3dbd18d5403559699558546996/diff",
"MergedDir": "/var/lib/docker/overlay2/ee03648cf34e03601848b1769569b4d3bb7192db118102ca050215ba87060bbf/merged",
"UpperDir": "/var/lib/docker/overlay2/ee03648cf34e03601848b1769569b4d3bb7192db118102ca050215ba87060bbf/diff",
"WorkDir": "/var/lib/docker/overlay2/ee03648cf34e03601848b1769569b4d3bb7192db118102ca050215ba87060bbf/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "volume",
"Name": "efk_esdata",
"Source": "/var/lib/docker/volumes/efk_esdata/_data",
"Destination": "/usr/share/elasticsearch/data",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "dab3a0ab0312",
"Domainname": "",
"User": "elasticsearch:root",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"ExposedPorts": {
"9200": {},
"9200/tcp": {},
"9300/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
