selfhosted-apps-docker/checkmk

checkmk

guide-by-example

Purpose

Monitoring of machines, containers, services, logs, ...

• Official site
• github

Monitoring in this case means gathering and showing information on how services or machines or containers are running. Can be cpu, io, ram, disk use, network throughput, latency,... can be number of http requests, errors, results of backups...

Overview

Good youtube overview.

checkmk is a fork of nagios and is mostly written in python.
Interesting fact is that there is no database where data are stored, RRD files for metrics and plaintext logs for everything else.

Agents are installed on machines that should be monitored, they expose gathered data at port 6556 for cmk to pull.
SNMP support as well.

Editions

Docs

• raw - 100% open source, unlimited use, some features are missing or are harder to set up. For example no push mode from agents.
• cloud - full featured with better performing version of the monitoring micro core, but with 750 services limit

I am gonna go with cloud for now, as 750 sounds like plenty for my use cases.

Files and directory structure

/home/
 └── ~/
     └── docker/
         └── checkmk/
             ├── 🗁 checkmk_data/
             ├── 🗋 docker-compose.yml
             └── 🗋 .env

• checkmk_data/ - a directory where checkmk_data stores its persistent data
• .env - a file containing environment variables for docker compose
• docker-compose.yml - a docker compose file, telling docker how to run the containers

The two files must be provided.
The directory is created by docker compose on the first run.

docker-compose

A simple compose.
Of note is use of ram as tmpfs mount into the container and setting a 1024 limit for max open files by a single process.

Note - the port is only expose, since theres expectation of use of a reverse proxy and accessing the services by hostname, not ip and port.

Docs on ports used in cmk.

docker-compose.yml

services:
  checkmk:
    # image: checkmk/check-mk-raw 
    image: checkmk/check-mk-cloud
    container_name: checkmk
    hostname: checkmk
    restart: unless-stopped
    env_file: .env
    ulimits:
      nofile: 1024
    tmpfs:
      - /opt/omd/sites/cmk/tmp:uid=1000,gid=1000
    volumes:
      - ./checkmk_data:/omd/sites
      - /etc/localtime:/etc/localtime:ro
    expose:
      - "5000"      # webgui
    ports:
      - 8000:8000   # agents who push

networks:
  default:
    name: $DOCKER_MY_NETWORK
    external: true

.env

# GENERAL
DOCKER_MY_NETWORK=caddy_net
TZ=Europe/Bratislava

# CMK
CMK_SITE_ID=dom
CMK_PASSWORD=WUx666yd0qCWh

All containers must be on the same network.
Which is named in the .env file.
If one does not exist yet: docker network create caddy_net

Reverse proxy

Caddy v2 is used, details here.

Caddyfile

cmk.{$MY_DOMAIN} {
  reverse_proxy checkmk:5000
}

First run

Visit cmk.example.com or whatever you set in reverse proxy.
Password for user cmkadmin is set the .env file.

Usual security recommendation is to create a new user and disable the default admin account.

Agents

Installation Windows Machine

Documentation

• Note the hostname and the ip address of the machine.

• Agent installation msi file is available at
  https://cmk.example.com/<site-name>/check_mk/agents/
  or webgui - Setup > Agents > Windows, Linux, Solaris, AIX > Windows - MSI
  downloads an msi, install.

• some win servers by default block ping by their firewall, allow ping through
  wf.msc - Inbound Rules - enable "File and Printer Sharing (Echo Request - ICMPv4-In)"

• CMK Web GUI > Setup > Hosts > Add host

  • Host name - <hostname> - should be all thats needed
  • IP address family - ipv4 only
  • IPv4 address - <ip address>

  Green button - Save & run service directory
  After a while list of services should be listed
  Top left green check mark - Accept all
  Yellow exclamation mark top right corner - to review changes
  Left top green button - Activate on selected sites

Agent registration - TLS

Will need password for user - agent_registration

• Setup > Users > agent_registration - edit - blue pencil left
• green dice - randomizes password - make the note of the new password
• apply changes - yellow exclamation mark

on the machine where the agent is installed

• cmd as administrator, not powershell
• cd "C:\Program Files (x86)\checkmk\service\"
• cmk-agent-ctl.exe register --hostname WIN-2022 --server cmk.example.com --site dom --user agent_registration --password "TJUE@ILTQFEUFQCT@ADS"
• DO MAKE SURE YOU USE THE CORRECT HOSTNAME
  I spent quite a while troubleshooting when I registered 3rd machine with hostname of the second machine that was already registered.

troubleshooting

• Setup > Hosts > <Host> > Save & run connection tests
• cmk-agent-ctl status - run on the host
• echo | nc 10.0.19.194 6556 - executed on the server, hosts ip is used
  before TLS it should reply with data, afterwards its 162%
• Monitor > Overview > Host search > 3 lines icon next to hostname > Download agent output

Installation Linux Machine

might be in repos, if not path to cmk instance has agents and plugins
https://cmk.example.com/<site-name>/check_mk/agents/

wget https://cmk.example.com/dom/check_mk/agents/check-mk-agent_2.3.0p6-1_all.deb sudo dpkg -i check-mk-agent_2.3.0p6-1_all.deb

docker plugin

wget https://cmk.example.com/dom/check_mk/agents/plugins/mk_docker.py
sudo install -m 0755 mk_docker.py /usr/lib/check_mk_agent/plugins

TLS

get password for user - agent_registration

sudo cmk-agent-ctl register --hostname debianu --server cmk.example.com --site dom --user agent_registration --password "TJUE@ILTQFEUFQCT@ADS"

troubleshooting

• sudo cmk-agent-ctl status
• ss -tulpn | grep 6556 - checks if the port is binded
• netstat -ano | grep 6556
• after reinstall the agent does not listen on the port
  sudo cmk-agent-ctl delete-all --enable-insecure-connections

SNMP monitoring

Using a mikrotik switch here

• login to mikrotik > IP > SNMP
• button Communities
  • disable public
  • Add New
  • Set Name - snmp_home
  • Security - Authorized
  • Read Access
  • authentication protocol - SHA1
  • Authentication Password - set some password
• back in SNMP settings
• enabled - checked
• Trap Community - snmp_home
• Trap Version - 3
• Apply

CMK Web GUI > Setup > Hosts > Add host

• host name - CRS310
• IPv4 address - 10.0.19.240
• Checkmk agent / API integrations - No API integrations, no Checkmk agent
• SNMP - SNMP v2 or v3
• SNMP credentials - Credentials for SNMPv3 with authentication but without privacy (authNoPriv)
• Authentication protocol - SHA1
• Security name - snmp_home
• Authentication Password - whatever was set
• Save and run service discovery
• apply changes

Push

Alerts

Logs

Update

Manual image update:

• docker-compose pull
  
• docker-compose up -d
  
• docker image prune

Backup and restore

Backup

Restore

• down the containers docker-compose down
  
• delete the entire monitoring directory
  
• from the backup copy back the monitoring directory
  
• start the containers docker-compose up -d