From aa93d18afdd03bd3e3d131fb459d561bdb87b276 Mon Sep 17 00:00:00 2001 From: DoTheEvo Date: Wed, 15 Mar 2023 00:41:04 +0100 Subject: [PATCH] update --- prometheus_grafana/readme.md | 375 ++++++++++++++++++++++++++++++----- 1 file changed, 322 insertions(+), 53 deletions(-) diff --git a/prometheus_grafana/readme.md b/prometheus_grafana/readme.md index bcb9c5c..351bda0 100644 --- a/prometheus_grafana/readme.md +++ b/prometheus_grafana/readme.md @@ -286,6 +286,20 @@ the default time interval is set to 1h instead of 15m ![interface-pic](https://i.imgur.com/wzwgBkp.png) +--- +--- + +# PromQL + +Some concept, highlights and examples of PromQL. + +PromQL returns results as vectors" + +* [The official](https://prometheus.io/docs/prometheus/latest/querying/basics/) basics page, quite to the point and short +* [relatively short video to the point](https://youtu.be/yLPTHinHB6Y) +* [decent stackoverflow answer](https://stackoverflow.com/questions/68223824/prometheus-instant-vector-vs-range-vector) + + --- --- @@ -573,21 +587,27 @@ has more detailed section on alerting worth checking out. ![loki_arch](https://i.imgur.com/aoMPrVV.png) -Loki is made by the grafana team. It's often refered as a Prometheus for logs.
-It is a **push** type monitoring, where an agent - **promtail** -pushes logs on to a Loki instance.
-For docker containers theres also an option to install **loki-docker-driver** -on a docker host and log pushing is set either globally in /etc/docker/daemon.json -or per container in compose files. +Loki is made by the grafana team. Sometimes called a Prometheus for logs, +it's a **push** type monitoring, where an agent - **promtail** +scrapes logs and then pushes them on to a Loki instance.
-There will be **two examples**.
+For docker containers theres also an option to install +[**loki-docker-driver**](https://grafana.com/docs/loki/latest/clients/docker-driver/) +on a docker host and log pushing is set either globally in /etc/docker/daemon.json +or per container in compose files.
+But as it turns out, **promtail** capabilities might be missed, +like its ability to add labels to logs it scrapes based on some rule. +Or processing data in some way, like translate IP addresses in to country +names or cities.
+Still loki-docker-driver is useful for getting containers logs in to loki +quickly and easily, with less cluttering of compose and less containers runnig. + +There will be **two examples** of logs monitoring.
A **minecraft server** and a **caddy revers proxy**, both docker containers. -## The setup +## Loki setup -To add Loki to the current stack: - -* New container - `loki` added to the compose file.
+* **New container** - `loki` added to the compose file.
Note the port 3100 is actually mapped to the host, allowing `localhost:3100` from driver to work. @@ -621,10 +641,10 @@ To add Loki to the current stack: ``` -* New file - `loki-docker-config.yml` bind mounted in the loki container.
+* **New file** - `loki-docker-config.yml` bind mounted in the loki container.
The file comes from [the official example](https://github.com/grafana/loki/tree/main/cmd/loki), - but url is changed, and compactor section is added, to have control over + but url is changed, and **compactor** section is added, to have control over [data retention.](https://grafana.com/docs/loki/latest/operations/storage/retention/)
@@ -675,43 +695,247 @@ To add Loki to the current stack: ```
-* Install [loki-docker-driver](https://grafana.com/docs/loki/latest/clients/docker-driver/)
- `docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions`
- To check if it's installed and enabled: `docker plugin ls` -* Containers that should be monitored need `logging` section in their compose.
+* #### loki-docker-driverdriver -
- docker-compose.yml + * **Install** [loki-docker-driver](https://grafana.com/docs/loki/latest/clients/docker-driver/)
+ `docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions`
+ To check if it's installed and enabled: `docker plugin ls`
+ * Containers that should be monitored usind loki-docker-driver need + `logging` section in their compose. - ```yml - services: +
+ docker-compose.yml - whoami: - image: "containous/whoami" - container_name: "whoami" - hostname: "whoami" - logging: - driver: "loki" - options: - loki-url: "http://localhost:3100/loki/api/v1/push" - ``` -
+ ```yml + services: -## Minecraft example + whoami: + image: "containous/whoami" + container_name: "whoami" + hostname: "whoami" + logging: + driver: "loki" + options: + loki-url: "http://localhost:3100/loki/api/v1/push" + ``` +
-Loki will be used to monitor logs of a [minecraft server.](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/minecraft)
-A dashboard will be created, showing logs volume in time.
-Alert will be set to send a notification when a player joins.
- -**Requirements** - grafana, loki, loki-docker-driver, minecraft with logging -set in compose +* #### promtail + + * Containers that should be monitored with **promtail** need it **added** + **to** their **compose** file, and made sure that it has access to the log files. + +
+ minecraft-docker-compose.yml + + ```yml + services: + + minecraft: + image: itzg/minecraft-server + container_name: minecraft + hostname: minecraft + restart: unless-stopped + env_file: .env + tty: true + stdin_open: true + ports: + - 25565:25565 # minecraft server players connect + volumes: + - ./minecraft_data:/data + + # LOG AGENT PUSHING LOGS TO LOKI + promtail: + image: grafana/promtail + container_name: minecraft-promtail + hostname: minecraft-promtail + restart: unless-stopped + volumes: + - ./minecraft_data/logs:/var/log/minecraft:ro + - ./promtail-config.yml:/etc/promtail-config.yml + command: + - '-config.file=/etc/promtail-config.yml' + + networks: + default: + name: $DOCKER_MY_NETWORK + external: true + ``` +
+ +
+ caddy-docker-compose.yml + + ```yml + services: + + caddy: + image: caddy + container_name: caddy + hostname: caddy + restart: unless-stopped + env_file: .env + ports: + - "80:80" + - "443:443" + - "443:443/udp" + volumes: + - ./Caddyfile:/etc/caddy/Caddyfile + - ./caddy_config:/data + - ./caddy_data:/config + - ./caddy_logs:/var/log/caddy + + # LOG AGENT PUSHING LOGS TO LOKI + promtail: + image: grafana/promtail + container_name: caddy-promtail + hostname: caddy-promtail + restart: unless-stopped + volumes: + - ./caddy_logs:/var/log/caddy:ro + - ./promtail-config.yml:/etc/promtail-config.yml + command: + - '-config.file=/etc/promtail-config.yml' + + networks: + default: + name: $DOCKER_MY_NETWORK + external: true + + ``` +
+ + * Generic **config file for promtail**, needs to be bind mounted + +
+ promtail-config.yml + + ```yml + clients: + - url: http://loki:3100/loki/api/v1/push + + scrape_configs: + - job_name: blablabla + static_configs: + - targets: + - localhost + labels: + job: blablabla_log + __path__: /var/log/blablabla/*.log + ``` +
+ +## Minecraft Loki example + +What can be seen in this example: + +* How to monitor logs of a docker container, + a [minecraft server](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/minecraft). +* How to visualize the logs in a dashboard. +* How to set an alert when a specific pattern appears in the logs. +* How to extract information from log to include it in the alert notification. +* Basic of grafana alert templates, so that the notification actually looks good, + and shows only relevant info. + +**Requirements** - grafana, loki, minecraft. ![logo](https://i.imgur.com/M1k0Dn4.png) -### First steps +### The Setup + +Initially **loki-docker-driver was used** to get logs to Loki, and it was simple +and worked nicely. But during alert stage +I **could not** figure out how to extract string from logs and include it in +an **alert** notification. Specificly to not just say that "a player joined", +but to have there name of the player that joined.
+The way to solve that was to **switch to promtail** and make use of its +[pipeline_stages](https://grafana.com/docs/loki/latest/clients/promtail/pipelines/). +Which was **suprisingly simple** and elegant. + +**Promtail** container is added to minecraft's **compose**, with bind mount +access to minecraf's logs.
+ +
+minecraft-docker-compose.yml + +```yml +services: + + minecraft: + image: itzg/minecraft-server + container_name: minecraft + hostname: minecraft + restart: unless-stopped + env_file: .env + tty: true + stdin_open: true + ports: + - 25565:25565 # minecraft server players connect + volumes: + - ./minecraft_data:/data + + # LOG AGENT PUSHING LOGS TO LOKI + promtail: + image: grafana/promtail + container_name: minecraft-promtail + hostname: minecraft-promtail + restart: unless-stopped + volumes: + - ./minecraft_data/logs:/var/log/minecraft:ro + - ./promtail-config.yml:/etc/promtail-config.yml + command: + - '-config.file=/etc/promtail-config.yml' + +networks: + default: + name: $DOCKER_MY_NETWORK + external: true +``` +
+ +**Promtail's config** is similar to the generic config in the previous section.
+The only addition is a short **pipeline** stage with **regex** that runs against +every log line before sending it to Loki. When matched **a label** `player` +is added to that log line. +The value of that label comes from the **named capture group** thats part of +that regex, the [syntax](https://www.regular-expressions.info/named.html) +is: `(?Pgroup)`
+This label will be easy to use later in the alert stage. + +
+promtail-config.yml + +```yml +clients: + - url: http://loki:3100/loki/api/v1/push + +scrape_configs: + - job_name: minecraft + static_configs: + - targets: + - localhost + labels: + job: minecraft_logs + __path__: /var/log/minecraft/*.log + pipeline_stages: + - regex: + expression: '.*:\s(?P.*)\sjoined the game$' + - labels: + player: +``` +
+ +[Here's regex101](https://regex101.com/r/5vkOU2/1) of it, +with some data to show how it works and bit of explanation. +[Here](https://stackoverflow.com/questions/74937454/how-to-add-custom-labels-in-promtail-config) +is stackoverflow answer that is the source for that config. + +![logo](https://i.imgur.com/LuUBZFn.png) + +### First steps in Grafana * In grafana, loki needs to be added as a datasource, `http://loki:3100` -* In `Explore` section, filter, container_name = minecraft, query... +* In `Explore` section, filter, job = minecraft_logs, Run query button... this should result in seeing minecraft logs and their volume/time graph. This Explore view will be recreated as a dashboard. @@ -752,9 +976,6 @@ When a player joins minecraft server a log appears *"Bastard joined the game"* - - Maybe one day I figure out how to pull player's name from the log - and pass it to alert, so far I got [this](https://regex101.com/r/pBAaEl/2) - `.*:\s(?P.*)\sjoined the game$` and [a full query](https://pastebin.com/Ep6PUwV2) - but dunno how to reference the named regex group in alert 4th section.
- And grafana forum is kinda big black hole of unanswared questions. + - Here is where the label `player` that was set in **promtail** is used
+ Summary = `{{ $labels.player }} joined the Minecraft server.` + - Can also pass values from expressions by targeting A/B/C/.. from step2
+ Description = `Number of players that joined in the last 5 min: {{ $values.B }}`
- **5 Notifications** - nothing - Save and exit -### Contact points +### Contact points - New contact point - Name = ntfy - Integration = Webhook - URL = https://ntfy.example.com/grafana + - Title = `{{ .CommonAnnotations.summary }}` + - Message = I put in [empty space unicode character](https://emptycharacter.com/) - Disable resolved message = check - Test - Save @@ -809,6 +1028,52 @@ and every one has to result in a a simple number or a true or false condition. After all this, there should be notification coming when a player joins. +### grafana-to-ntfy + +For alerts one can use +[ntfy](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/gotify-ntfy-signal) +but on its own the alerts from grafana are just plain text json.
+[Here's](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/gotify-ntfy-signal#grafana-to-ntfy) +how to setup grafana-to-ntfy, to make the alerts look good. + +![ntfy](https://i.imgur.com/gL81jRg.png) + +### Templates + +Not really used here, but heres some basics as it took embarasignly long +to find that `{{ .CommonAnnotations.summary }}` for the title. + +
+

Templates basic

 + +* Testing should be done in contact point when editing, + useful Test button that allows you send alerts with custom values. +* My big mistake when playing with this was missing a dot. + In Contact point, in Title/Message input box. + * correct one - `{{ template "test" . }}` + * the one I had - `{{ template "test" }}`
+* So yeah, dot is important in here. It represents data and context + passed to a template. It can represent global context or when used inside + `{{ range }}` it represents iteration loop value, like `i` in classic for loop. +* [This](https://pastebin.com/id3264k6) json structure is what an alert looks + like. Notice `alerts` being an array and `commonAnnotations` being object. + If something is an array, theres need to loop over it to get acces to the + values in it. For objects one just needs to target the value. +* To [iterate over alerts array.](https://i.imgur.com/gdwGhjN.png) +* To just access a value - `{{ .CommonAnnotations.summary }}` + +Templates resources + +* [Overview of Grafana Alerting and Message Templating for Slack](https://faun.pub/overview-of-grafana-alerting-and-message-templating-for-slack-6bb740ec44af) +* [youtube - Unified Alerting Grafana 8 | Prometheus | Victoria | Telegraf | Notifications | Alert Templating](https://youtu.be/UtmmhLraSnE) +* [Dot notation](https://www.practical-go-lessons.com/chap-32-templates#dot-notation) +* + +
+ +--- +--- + # Caddy monitoring Reverse proxy is kinda linchpin of a selfhosted setup, since it's in charge @@ -1107,6 +1372,10 @@ and push them to Loki. Once there, a basic grafana dashboard can be made. * Deduplication - Signature * Save +useful resources + +* https://www.youtube.com/watch?v=UtmmhLraSnE + ## Geoip [to-do](https://github.com/grafana/loki/blob/main/docs/sources/clients/promtail/stages/geoip.md)