How to Install and Monitor Servers with Checkmk on Ubuntu 22.04

Checkmk is a server and application monitoring software written in Python and C++. It supports the monitoring of servers, applications, networks, containers, and clouds. It has a wide range of features, including, automated monitoring, agentless monitoring via HTTP/SNMP, over 1900 plugins to collect data, detailed network traffic analysis, customizable dashboards, and Grafana support.

Checkmk comes in four editions. The first version is the open-source Raw edition and uses Nagios as its core. The second version is the free version which has all the features of the standard edition, supports unlimited hosts for the first 30 days, and afterward is limited to 25 hosts. The enterprise standard edition and the enterprise-managed editions are advanced versions with support for unlimited hosts and multiple customers.

In this tutorial, you will learn how to install and monitor servers using Checkmk on a Ubuntu 22.04 machine.

Prerequisites

A server running Ubuntu 22.04. The Standard edition will give a warning if the number of CPU cores on your server is less than 4. However, you can safely ignore this warning but depending upon the number of hosts you want to monitor, you should opt for a more powerful machine. The Raw edition doesn't perform such a check.
A non-root user with sudo privileges.
The uncomplicated Firewall(UFW) is enabled and running.
A Fully Qualified domain name (FQDN) pointing to the server like, checkmk.example.com.
Everything is updated.
```
$ sudo apt update && sudo apt upgrade
```

Step 1 - Configure Firewall

Before installing any packages, the first step is configuring the firewall to allow HTTP and HTTPS connections.

Check the status of the firewall.

$ sudo ufw status

You should see something like the following.

Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)

Allow HTTP and HTTPs ports.

$ sudo ufw allow http
$ sudo ufw allow https

Check the status again to confirm.

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443                        ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)

Step 2 - Install Checkmk

For our tutorial, we will be working with the Standard Free edition of Checkmk. You can however use any edition you want. The commands will remain the same.

The first step is to grab the GPG key for Checkmk.

$ wget https://download.checkmk.com/checkmk/Check_MK-pubkey.gpg

Import the key.

$ gpg --import Check_MK-pubkey.gpg

At the time of writing this tutorial, 2.1.0p11 is the latest available version. You can check the official Checkmk download page to grab whichever version/edition of Checkmk you want. Copy the command of the version and edition you want. Since we are installing the Standard edition, our download command will look like the following.

$ wget https://download.checkmk.com/checkmk/2.1.0p11/check-mk-free-2.1.0p11_0.jammy_amd64.deb

Install the Checkmk package.

$ sudo apt install ./check-mk-free-2.1.0p11_0.jammy_amd64.deb

The Checkmk package will install the Apache webserver for you. If you have an existing server on your machine, you can configure it to work as a proxy for Apache.

Confirm whether Checkmk has been installed.

$ omd version
OMD - Open Monitoring Distribution Version 2.1.0p11.cfe

The Checkmk monitoring system uses the Open Monitoring Distribution (OMD) project which is managed via the command line using the omd command.

Step 3 - Create Checkmk Site

OMD tool is used to create and manage multiple monitoring sites on a single server. Each site is a self-contained monitoring system running independently of others.

Run the following command to create the Checkmk site.

$ sudo omd create howtoforge

You will get a similar output.

Adding /opt/omd/sites/howtoforge/tmp to /etc/fstab.
Creating temporary filesystem /omd/sites/howtoforge/tmp...OK
Updating core configuration...
Generating configuration for core (type cmc)...
Starting full compilation for all hosts Creating global helper config...OK
 Creating cmc protobuf configuration...OK
Executing post-create script "01_create-sample-config.py"...OK
Restarting Apache...OK
Created new site howtoforge with version 2.1.0p11.cfe.

  The site can be started with omd start howtoforge.
  The default web UI is available at http://checkmk/howtoforge/

  The admin user for the web applications is cmkadmin with password: 9sWdQemE
  For command line administration of the site, log in with 'omd su howtoforge'.
  After logging in, you can change the password for cmkadmin with 'htpasswd etc/htpasswd cmkadmin'.

The above command performs the following steps.

A system user and a group named howtoforge is created.
A home directory /omd/sites/howtoforge is created and assigned to the above user. This is called the site directory.
The home directory is populated with configuration files and sub-directories.
A basic configuration file is created for the site.
Apache server is configured and restarted.

You can perform the site operations after logging in to the howtoforge user using the following command.

$ sudo omd su howtoforge

This will switch you to its login shell.

OMD[howtoforge]:-$

Alternatively, you can perform the site operations using sudo keyword.

Start the site.

OMD[howtoforge]:-$ omd start
Temporary filesystem already mounted
Starting agent-receiver...OK
Starting mkeventd...OK
Starting liveproxyd...OK
Starting mknotifyd...OK
Starting rrdcached...OK
Starting cmc...OK
Starting apache...OK
Starting dcd...OK
Starting redis...OK
Initializing Crontab...OK

Use the following command to stop the site.

OMD[howtoforge]:-$ omd stop

Check the site's status using the following command.

OMD[mysite]:~$ omd status
agent-receiver: running
mkeventd:       running
liveproxyd:     running
mknotifyd:      running
rrdcached:      running
cmc:            running
apache:         running
dcd:            running
redis:          running
crontab:        running
-----------------------
Overall state:  running

To change the default password for Checkmk, issue the following command.

OMD[howtoforge]:-$ htpasswd -m etc/htpasswd cmkadmin
New password:
Re-type new password:
Updating password for user cmkadmin

Step 4 - Access Checkmk site

Visit the URL http://<serverIP>/howtoforge or http://checkmk.example.com/howtoforge and you will get the following login page. Log in using the credentials provided in the previous step to log in to the monitoring site.

Checkmk Login Page

Once logged in, you will see the Checkmk dashboard.

Checkmk Dashboard

Step 5 - Install SSL

We need to install Certbot to generate the SSL certificate. We will install Certbot using Snapd.

Ubuntu 22.04 comes with Snapd installed by default. Run the following commands to ensure that your version of Snapd is up to date.

$ sudo snap install core
$ sudo snap refresh core

Install Certbot.

$ sudo snap install --classic certbot

Use the following command to ensure that the Certbot command can be run by creating a symbolic link to the /usr/bin directory.

$ sudo ln -s /snap/bin/certbot /usr/bin/certbot

Run the following command to generate an SSL Certificate.

$ sudo certbot certonly --apache --agree-tos --no-eff-email --staple-ocsp --preferred-challenges http -m [email protected] -d checkmk.example.com

The above command will download a certificate to the /etc/letsencrypt/live/checkmk.example.com directory on your server.

To check whether the SSL renewal is working fine, do a dry run of the process.

$ sudo certbot renew --dry-run

If you see no errors, you are all set. Your certificate will renew automatically.

Step 6 - Configure Apache Server

Enable the Apache headers module.

$ sudo a2enmod headers

Enable the mod_ssl apache module.

$ sudo a2enmod ssl

Enable the HTTP/2 module.

$ sudo a2enmod http2

Create and open the file /etc/apache2/sites-enabled/howtoforge.conf for editing.

Paste the following code in it.

<VirtualHost *:80>
    ServerName checkmk.example.com
    RewriteEngine On
    RewriteCond %{SERVER_PORT} !^443$
    RewriteRule (.*) https://%{HTTP_HOST}$1 [L]
    RequestHeader set X-Forwarded-Proto "https"
</VirtualHost>

<VirtualHost *:443>
    ServerName checkmk.example.com
    ErrorLog ${APACHE_LOG_DIR}/checkmk.error.log
    CustomLog ${APACHE_LOG_DIR}/checkmk.access.log combined
    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/checkmk.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/checkmk.example.com/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/checkmk.example.com/chain.pem
    Protocols h2 http/1.1
</VirtualHost>

Save the file by pressing Ctrl + X and entering Y when prompted.

Create and open the file /etc/apache2/conf-available/ssl-params.conf for editing. Paste the following code in it.

SSLProtocol -all +TLSv1.3 +TLSv1.2
SSLOpenSSLConfCmd Curves X25519:secp521r1:secp384r1:prime256v1
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder on
SSLSessionTickets off
SSLCompression off

SSLUseStapling On
SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"
SSLStaplingResponseMaxAge 900

Save the file by pressing Ctrl + X and entering Y when prompted.

Deactivate the default site.

$ sudo a2dissite 000-default.conf

Activate the Checkmk site.

$ sudo a2ensite howtoforge.conf

Activate the SSL configuration file.

$ sudo a2enconf ssl-params

Verify the configuration.

$ sudo apache2ctl configtest

If you get OK as the response, proceed with restarting the Apache server to activate the Checkmk site.

$ sudo systemctl restart apache2

You can now access your Checkmk site via the URL https://checkmk.example.com/howtoforge.

Step 7 - Install Monitoring Agent

You need to install a monitoring agent to add and monitor other hosts in Checkmk. To install the agent, open the Checkmk site, and go to Setup > Agents > Windows, Linux, Solaris, AIX option.

Checkmk Agents Menu

You will be taken to the following page.

Checkmk Agents Download Page

Click the Related menu item on the top and then select Linux, Solaris, AIX files from the dropdown menu.

Checkmk Linux Files Menu

Copy the download link to the deb file from the following page.

Checkmk Agents Download Package Page

Download the agent using the copied link onto the host server.

$ wget https://checkmk.example.com/howtoforge/check_mk/agents/check-mk-agent_2.1.0p11-1_all.deb

Install the downloaded package.

$ sudo apt install ./check-mk-agent_2.1.0p11-1_all.deb

Verify if the agent is installed.

$ check_mk_agent

You will get a similar output.

$ check_mk_agent
<<<check_mk>>>
Version: 2.1.0p11
AgentOS: linux
Hostname: checkmk
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
.......

Before proceeding ahead, we need to open port 6556 which is used by the server to connect to the Checkmk host.

$ sudo ufw allow 6556

The next step is to register the host with the server. This step is necessary for a secured Checkmk server. You can skip it if you don't use SSL on your server.

There are some bugs with the latest agent package, therefore, run the following commands as a workaround.

$ sudo /var/lib/cmk-agent/scripts/cmk-agent-useradd.sh
$ sudo /var/lib/cmk-agent/scripts/super-server/setup trigger

Restart the following services.

$ sudo systemctl restart cmk-agent-ctl-daemon.service --now
$ sudo systemctl restart check-mk-agent.socket --now

Issue the following command to register the host with your Checkmk server.

$  sudo cmk-agent-ctl register --hostname localhost --server checkmk.example.com:443 --site howtoforge --user cmkadmin

You will get a similar output where you will be asked if you want to establish a connection and the password for the Checkmk server.

Attempting to register at checkmk.nspeaks.xyz:443/howtoforge. Server certificate details:

PEM-encoded certificate:
-----BEGIN CERTIFICATE-----
MIIFLTCC...........
.............udEkKI
-----END CERTIFICATE-----

Issued by:
        R3
Issued to:
        checkmk.example.com
Validity:
        From Mon, 05 Sep 2022 08:12:29 +0000
        To   Sun, 04 Dec 2022 08:12:28 +0000

Do you want to establish this connection? [Y/n]
> y

Please enter password for 'cmkadmin'
>
ERROR [cmk_agent_ctl] Error pairing with checkmk.nspeaks.xyz:443/howtoforge

Caused by:
    Request failed with code 404 Not Found: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html><head>
    <title>404 Not Found</title>
    </head><body>
    <h1>Not Found</h1>
    <p>The requested URL was not found on this server.</p>
    </body></html>

You will probably see an error at the end of the command. This is due to the buggy Checkmk agent package at the time of writing this tutorial. But the host is registered successfully with the server.

Step 8 - Add Host for Monitoring

To add your host, visit the Setup option from the left panel and then open the Hosts > Hosts and click Add Host button to proceed.

Checkmk Add Host

Enter the hostname and the IP address of the host server, and click the Save & go to service configuration button.

Add Checkmk Host Page

On the services page, you will see a list of all discovered services. If you see any Undecided services, click the Fix all link to fix the issue.

Checkmk Discovered Services

Then, click the yellow-colored sign on the top right corner of the window.

Checkmk Pending Changes

Next, click the Activate on selected sites button.

> How to Install and Monitor Servers with Checkmk on Ubuntu 22.04