diff --git a/prometheus_grafana/readme.md b/prometheus_grafana/readme.md index 658a9a8..52563e9 100644 --- a/prometheus_grafana/readme.md +++ b/prometheus_grafana/readme.md @@ -4,12 +4,15 @@ ![logo](https://i.imgur.com/e03aF8d.png) +WORK IN PROGRESS
+Loki and caddy monitoring parts are not finished yet + # Purpose Monitoring of the host and the running cointaners. * [Official Prometheus site](https://prometheus.io/) -* [Github](https://github.com/prometheus) +* [Official Grafana site](https://grafana.com/) Most of the stuff here is based off the magnificent [stefanprodan/dockprom.](https://github.com/stefanprodan/dockprom)
@@ -17,13 +20,11 @@ So maybe just go play with that. # Chapters -Setup here starts off with the basics and then theres chapters how to add features - * **[Core prometheus+grafana](#Overview)** - to get nice dashboards with metrics from docker host and containers * **[Pushgateway](#Pushgateway)** - how to use it to allow pushing metrics in to prometheus from anywhere * **[Alertmanager](#Alertmanager)** - how to use it for notifications -* **Loki** - how to do the above things but for logs, not just metrics -* **Caddy** - adding dashboard for reverse proxy info +* **[Loki](#Loki)** - how to do the above things but for logs, not just metrics +* **[Caddy monitoring](#Caddy_monitoring)** - adding dashboard for reverse proxy info # Overview @@ -110,8 +111,8 @@ services: image: prom/prometheus:v2.42.0 container_name: prometheus hostname: prometheus - restart: unless-stopped user: root + restart: unless-stopped depends_on: - cadvisor command: @@ -134,9 +135,9 @@ services: image: grafana/grafana:9.3.6 container_name: grafana hostname: grafana + user: root restart: unless-stopped env_file: .env - user: root volumes: - ./grafana_data:/var/lib/grafana expose: @@ -320,8 +321,8 @@ Including pushing information from windows powershell. image: prom/prometheus:v2.42.0 container_name: prometheus hostname: prometheus - restart: unless-stopped user: root + restart: unless-stopped depends_on: - cadvisor command: @@ -345,9 +346,9 @@ Including pushing information from windows powershell. image: grafana/grafana:9.3.6 container_name: grafana hostname: grafana + user: root restart: unless-stopped env_file: .env - user: root volumes: - ./grafana_data:/var/lib/grafana expose: @@ -507,16 +508,27 @@ Including pushing information from windows powershell.

Loki

+ Loki is made by grafana team, who call it Prometheus for logs.
+ It is a **push** type monitoring, where an agent(**promtail**) + pushes logs on to a Loki instance. + + What this example will set out to do is monitor logs of a + [minecraft server.](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/minecraft) + Have some dashboards about logs and have an alert when a player joins. + + To add to the existing setup: + - New container - `loki` added to the compose file. It stores logs and makes them available for grafana to visualize. - New container - `promtail` added to the compose file. Agent that pushes - logs content to loki. + logs content to loki. This container needs access to logs to be pushed + and so docker host `/var/lib/docker/containers` directory is bind mounted. - New file - `loki-docker-config.yml` bind mounted in the loki container.
The file is [all default](https://github.com/grafana/loki/tree/main/cmd/loki), except for alertmanager url.
- New file - `promtail-config.yml` mounted in to promtail container
- This files defines source of logs. Any such source much be accessible - by the promtail container. + This files defines source of logs. In here a minecraft container is set, + with some added labels for easier targeting of logs.
docker-compose.yml @@ -529,8 +541,8 @@ Including pushing information from windows powershell. image: prom/prometheus:v2.42.0 container_name: prometheus hostname: prometheus - restart: unless-stopped user: root + restart: unless-stopped depends_on: - cadvisor command: @@ -554,9 +566,9 @@ Including pushing information from windows powershell. image: grafana/grafana:9.3.6 container_name: grafana hostname: grafana + user: root restart: unless-stopped env_file: .env - user: root volumes: - ./grafana_data:/var/lib/grafana expose: @@ -626,6 +638,7 @@ Including pushing information from windows powershell. image: grafana/loki:2.7.3 container_name: loki hostname: loki + user: root restart: unless-stopped volumes: - ./loki_data:/loki @@ -642,8 +655,8 @@ Including pushing information from windows powershell. image: grafana/promtail:2.7.3 container_name: promtail hostname: promtail - restart: unless-stopped user: root + restart: unless-stopped volumes: - /var/log:/var/log:ro - /var/lib/docker/containers:/var/lib/docker/containers:ro @@ -724,8 +737,28 @@ Including pushing information from windows powershell. ```
+There was also an option to use loki logging driver to just get docker containers. +But any container re-recreated with that setting set took always long time to down.
+ +Anyway, now with these files in place.. + +* Loki needs to be added to in grafana as a datasource.
+ If everything works as it should, there should be no notice, down left side:
+ `Data source connected and labels found.` +
+--- +--- + +
+

Caddy monitoring

+ +
+ +--- +--- + # Update Manual image update: