# Caddy v2 Reverse Proxy ###### guide-by-example ![logo](https://i.imgur.com/xmSY5qu.png) 1. [Purpose & Overview](#Purpose--Overview) 2. [Caddy as a reverse proxy in docker](#Caddy-as-a-reverse-proxy-in-docker) 3. [Caddy more info and various configurations](#Caddy-more-info-and-various-configurations) 4. [Caddy DNS challenge](#Caddy-DNS-challenge) 5. [Other guides](#Other-guides) *[Older version](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/d973916d56e23bb5564bd9b68e06ec884cfc6af1/caddy_v2) of this guide - more detailed and handholding for docker noobs.* # Purpose & Overview Reverse proxy is needed if one wants access to services based on the hostname.
For example `nextcloud.example.com` points traffic to nextcloud docker container, while `jellyfin.example.com` points to the media server on the network. * [Official site](https://caddyserver.com/v2) * [Official documentation](https://caddyserver.com/docs/) * [Forum](https://caddy.community/) * [Github](https://github.com/caddyserver/caddy) Caddy is a pretty damn good web server with automatic HTTPS. Written in Go. Web servers are build to deal with http traffic, so they are an obvious choice for the function of reverse proxy. In this setup Caddy is used mostly as [a TLS termination proxy](https://www.youtube.com/watch?v=H0bkLsUe3no). Https encrypted tunel ends with it, so that the traffic can be analyzed and send to a correct webserver based on the settings in `Caddyfile`. Caddy with its build-in automatic https allows configs to be clean and simple and to just work. ``` nextcloud.example.com { reverse_proxy nextcloud-web:80 } jellyfin.example.com { reverse_proxy 192.168.1.20:80 } ``` And **just works** means fully works. No additional configuration needed for https redirect, or special services if target is not a container, or need to deal with load balancer, or need to add boilerplate headers for x-forward, or other extra work.
It has great out of the box defaults, fitting majority of uses and only some special casess with extra functionality need extra work. ![url](https://i.imgur.com/iTjzPc0.png) # Caddy as a reverse proxy in docker Caddy will be running as a docker container and will route traffic to other containers, or machines on the network. ### - Files and directory structure ``` /home/ └── ~/ └── docker/ └── caddy/ ├── 🗁 caddy_config/ ├── 🗁 caddy_data/ ├── 🗋 .env ├── 🗋 Caddyfile └── 🗋 docker-compose.yml ``` * `caddy_config/` - a directory containing configs that Caddy generates, most notably `autosave.json` which is a backup of the last loaded config * `caddy_data/` - a directory storing TLS certificates * `.env` - a file containing environment variables for docker compose * `Caddyfile` - the Caddy configuration file * `docker-compose.yml` - a docker compose file, telling docker how to run containers You only need to provide the three files.
The directories are created by docker compose on the first run, the content of these is visible only as root of the docker host. ### - Create a new docker network `docker network create caddy_net` All the future containers and Caddy must be on the same network, ping-able by their hostnames. ### - Create docker-compose.yml and .env file Basic simple docker compose, using the official caddy image.
Ports 80 and 443 are pusblished/mapped on to docker host as Caddy is the one in charge of any traffic coming there.
`docker-compose.yml` ```yml services: caddy: image: caddy container_name: caddy hostname: caddy restart: unless-stopped env_file: .env ports: - "80:80" - "443:443" - "443:443/udp" volumes: - ./Caddyfile:/etc/caddy/Caddyfile - ./caddy_config:/data - ./caddy_data:/config networks: default: name: $DOCKER_MY_NETWORK external: true ``` `.env` ```php # GENERAL TZ=Europe/Bratislava DOCKER_MY_NETWORK=caddy_net MY_DOMAIN=example.com ``` You obviously want to change `example.com` to your domain. ### - Create Caddyfile `Caddyfile` ``` a.{$MY_DOMAIN} { reverse_proxy whoami:80 } b.{$MY_DOMAIN} { reverse_proxy nginx:80 } ``` `a` and `b` are the subdomains, can be named whatever.
For them to work they **must have type-A DNS record set**, that points at your public ip set on Cloudflare, or wherever the domains DNS is managed.
Can test if correctly set with online dns lookup tools, [like this one.](https://mxtoolbox.com/DNSLookup.aspx) The value of `{$MY_DOMAIN}` is provided by the `.env` file.
The subdomains point at docker containers by their **hostname** and **exposed port**. So every docker container you spin should have hostname definied and be on `caddy_net`, or some other named custom network, as the default bridge docker network [does not provide](https://docs.docker.com/network/bridge/) automatic DNS resolution between containers.

Setup some docker containers

 Something light to setup to route to that has a webpage to show.
Not bothering with an `.env` file here. Note the lack of published/mapped ports in the compose, as they will be accessed only through Caddy, which has it's ports published.
Containers on the same bridge docker network can access each other on any port.
*extra info:*
To know which ports containers have exposed - `docker ps`, or `docker port `, or use [ctop](https://github.com/bcicen/ctop). `whoami-compose.yml` ```yaml services: whoami: image: "containous/whoami" container_name: "whoami" hostname: "whoami" networks: default: name: caddy_net external: true ``` `nginx-compose.yml` ```yaml services: nginx: image: nginx:latest container_name: nginx hostname: nginx networks: default: name: caddy_net external: true ```
--- ---

Editing hosts file

 If the docker host is with you on your local network then you need to deal with bit of an issue. When you write that `a.example.com` in to your browser, you are asking internet DNS server for IP address of `a.example.com`. DNS servers will reply with your own public IP, and most consumer routers wont allow this loopback, where your requests should go out and then right back. So just [edit](https://support.rackspace.com/how-to/modify-your-hosts-file/) `hosts` as root/administrator, adding whatever is the local IP of the docker host and the hostname: ``` 192.168.1.222 a.example.com 192.168.1.222 b.example.com ``` You can test what are the replies for DNS requests with the command `nslookup a.example.com`, works in linux and windows. If it is just quick testing one can use Opera browser and enable its build in VPN.
This edit of a host file works only on that one machine. To solve it for all devices theres need to to run dns server on the network, or running a higher tier firewall/router. * [Here's](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/dnsmasq) a guide-by-example for dnsmasq. * [Here's](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/opnsense) a guide-by-example for opnsense firewall
--- --- ### - Run it all Run all the containers. Give Caddy time to get certificates, checking `docker logs caddy` as it goes, then visit the urls. It should lead to the services with https working. If something is fucky use `docker logs caddy` to see what is happening.
Restarting the container `docker container restart caddy` can help. Or investigate inside `docker exec -it caddy /bin/sh`. For example trying to ping hosts that are suppose to be reachable, `ping nginx` should work. There's also other possible issues, like bad port forwarding towards docker host, or ISP not providing you with publicly reachable IP. *extra info:*
`docker exec -w /etc/caddy caddy caddy reload` reloads config if you made changes and want them to take effect. *extra info2:*
caddy can complain about formatting of the `Caddyfile`
this executed on the host will let caddy overwrite the Caddyfile with correct formatting `docker exec -w /etc/caddy caddy caddy fmt -overwrite` # Caddy more info and various configurations ##### Caddyfile structure: ![caddyfile-diagram-pic](https://i.imgur.com/c0ycNal.png) Worth having a look at the official documentation, especially these short pages * [concept](https://caddyserver.com/docs/caddyfile/concepts) * [conventions](https://caddyserver.com/docs/conventions) * [reverse_proxy](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy) Maybe checking out [mozzila's - overview of HTTP](https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview) would also not hurt, it is very well written. ### Routing traffic to other machines on the LAN If not targeting a docker container but a dedicated machine on the network.
Nothing really changes, if you can ping the machine from Caddy container by its hostname or its IP, it will work. ``` blue.{$MY_DOMAIN} { reverse_proxy server-blue:80 } violet.{$MY_DOMAIN} { reverse_proxy 192.168.1.100:80 } ``` ### Redirect Here is an example of a redirect for the common case of switching anyone that comes to `www.example.com` to the naked domain `example.com`. ```php www.{$MY_DOMAIN} { redir https://{$MY_DOMAIN}{uri} } ``` Or what if theres a need for a short url for something often used, but selfhosted url-shorterners seem bloate... looking at you Shlink and Kutt. ```php down.{$MY_DOMAIN} { redir https://nextcloud.example.com/s/CqJyOijYeezESQT/download } ``` or if prefering doing path instead of subdomain ```php {$MY_DOMAIN} { reverse_proxy whoami:80 redir /down https://nextcloud.example.com/s/CqJyOijYeezESQT/download } ``` Another example is running NextCloud behind proxy, which likely shows few warning on its status page. These require some redirects for service discovery to work and would like if [HSTS](https://www.youtube.com/watch?v=kYhMnw4aJTw) [2](https://www.youtube.com/watch?v=-MWqSD2_37E) would be set.
Like so: ```php nextcloud.{$MY_DOMAIN} { reverse_proxy nextcloud:80 header Strict-Transport-Security max-age=31536000; redir /.well-known/carddav /remote.php/carddav 301 redir /.well-known/caldav /remote.php/caldav 301 } ``` ### Named matchers and IP filtering Caddy has [matchers](https://caddyserver.com/docs/caddyfile/matchers)
* `*` to match all requests (wildcard; default). * `/path` start with a forward slash to match a request path. * `@name` to specify a named matcher. In `reverse_proxy server-blue:80` matcher is ommited and in that case the default - `*` applies meaning all traffic. But if more control is desired, path matchers and named matchers come to play. What if all traffic coming from the outside world should be blocked, but local network be allowed through?
Well, the [remote_ip](https://caddyserver.com/docs/caddyfile/matchers#remote-ip) matcher comes to play, which enables you to filter requests by their IP.
Named matchers are defined by `@` and can be named whatever you like. ``` { # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory } a.{$MY_DOMAIN} { reverse_proxy whoami:80 } b.{$MY_DOMAIN} { reverse_proxy nginx:80 @fuck_off_world { not remote_ip 192.168.1.0/24 } respond @fuck_off_world 403 } ``` `@fuck_off_world` matches all IPs except the local network IP range.
Requests matching that rule get the response 403 - forbidden. ### Snippets What if you need to have the same matcher in several site-blocks and would prefer for config to look cleaner? Here comes the [snippets](https://caddyserver.com/docs/caddyfile/concepts#snippets).
Snippets are defined under the global options block, using parentheses, named whatever you like.
They then can be used inside any site-block with simple `import ` Now would be a good time to look again at that concept picture above. Here is above example of IP filtering named matcher done using a snippet. ``` { # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory } (LAN_only) { @fuck_off_world { not remote_ip 192.168.1.0/24 } respond @fuck_off_world 403 } a.{$MY_DOMAIN} { reverse_proxy whoami:80 } b.{$MY_DOMAIN} { reverse_proxy nginx:80 import LAN_only } ``` ### Backend communication Some containers might be set to communicate only through https 443 port. But since they are behind proxy, their certificates wont be singed, wont be trusted. Caddies sub-directive `transport` sets how to communicate with the backend.
Setting the upstream's scheme to `https://` or declaring the `tls` transport subdirective makes it use https. Setting `tls_insecure_skip_verify` makes Caddy ignore errors due to untrusted certificates coming from the backend. ``` whatever.{$MY_DOMAIN} { reverse_proxy https://server-blue:443 { transport http { tls tls_insecure_skip_verify } } } ``` ### Headers and gzip This example is with vaultwarden password manager, which comes with its reverse proxy [recommendations](https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples). `encode gzip` enables compression.
This lowers the bandwith use and speeds up loading of the sites. It is often set on the webserver running inside the docker container, but if not it can be enabled on caddy. You can check if your stuff has it enabled by using one of [many online tools](https://varvy.com/tools/gzip/) By default, Caddy passes through Host header and adds X-Forwarded-For for the client IP. This means that 90% of the time a simple config is all that is needed but sometimes some extra headers might be desired. Here we see vaultwarden make use of some extra headers.
We can also see its use of websocket protocol for notifications at port 3012. ``` vault.{$MY_DOMAIN} { encode gzip header { # Enable cross-site filter (XSS) and tell browser to block detected attacks X-XSS-Protection "1; mode=block" # Disallow the site to be rendered within a frame (clickjacking protection) X-Frame-Options "DENY" # Prevent search engines from indexing (optional) X-Robots-Tag "none" # Server name removing -Server } # Notifications redirected to the websockets server reverse_proxy /notifications/hub vaultwarden:3012 # Proxy the Root directory to Rocket reverse_proxy vaultwarden:80 } ``` ### Basic authentication [Official documentation.](https://caddyserver.com/docs/caddyfile/directives/basicauth)
Directive `basicauth` can be used when one needs to add a username/password check before accessing a service. Password is [bcrypt](https://www.devglan.com/online-tools/bcrypt-hash-generator) hashed and then [base64](https://www.base64encode.org/) encoded.
You can use the [`caddy hash-password`](https://caddyserver.com/docs/command-line#caddy-hash-password) command to hash passwords for use in the config. Config bellow has login/password : `bastard`/`bastard` `Caddyfile` ``` b.{$MY_DOMAIN} { reverse_proxy whoami:80 basicauth { bastard JDJhJDA0JDVkeTFJa1VjS3pHU3VHQ2ZSZ0pGMU9FeWdNcUd0Wk9RdWdzSzdXUXNhWFFLWW5pYkxXVEU2 } } ``` ### Logging [Official documentation.](https://caddyserver.com/docs/caddyfile/directives/log)
If access logs for specific site are desired ``` bookstack.{$MY_DOMAIN} { log { output file /data/logs/bookstack_access.log { roll_size 20mb roll_keep 5 } } reverse_proxy bookstack:80 } ``` # Caddy DNS challenge This setup only works for Cloudflare. DNS challenge authenticates ownership of the domain by requesting that the owner puts a specific TXT record in to the domains DNS zone.
Benefit of using DNS challenge is that there is no need for your server to be reachable by the letsencrypt servers. Cant open ports or want to exclude entire world except your own country from being able to reach your server? DNS challange is what you want to use for https then.
It also allows for issuance of wildcard certificates.
The drawback is a potential security issue, since you are creating a token that allows full control over your domain's DNS. You store this token somewhere, you are giving it to some application from dockerhub... *note*: caddy uses a new [libdns](https://github.com/libdns/libdns/) golang library with [cloudflare package](https://github.com/libdns/cloudflare) ### - Create API token on Cloudflare [On Cloudflare](https://dash.cloudflare.com/profile/api-tokens) create a new API Token with two permsisions, [pic of it here](https://i.imgur.com/YWxgUiO.png) * zone/zone/read
* zone/dns/edit
Include all zones needs to be set. ### - Edit .env file Add `CLOUDFLARE_API_TOKEN` variable with the value of the newly created token. `.env` ``` MY_DOMAIN=example.com DOCKER_MY_NETWORK=caddy_net CLOUDFLARE_API_TOKEN= ``` ### - Create Dockerfile To add support, Caddy needs to be compiled with [Cloudflare DNS plugin](https://github.com/caddy-dns/cloudflare).
This is done by using your own Dockerfile, using the `builder` image. Create a directory `dockerfile-caddy` in the caddy directory.
Inside create a file named `Dockerfile`. `Dockerfile` ```Dockerfile FROM caddy:2.6.2-builder AS builder RUN xcaddy build \ --with github.com/caddy-dns/cloudflare FROM caddy:2.6.2 COPY --from=builder /usr/bin/caddy /usr/bin/caddy ``` ### - Edit docker-compose.yml `image` replaced with `build` option pointing at the `Dockerfile` location
and `CLOUDFLARE_API_TOKEN` variable added. `docker-compose.yml` ```yml services: caddy: build: ./dockerfile-caddy container_name: caddy hostname: caddy restart: unless-stopped ports: - "80:80" - "443:443" environment: - MY_DOMAIN - CLOUDFLARE_API_TOKEN volumes: - ./Caddyfile:/etc/caddy/Caddyfile:ro - ./caddy_data:/data - ./caddy_config:/config networks: default: name: $DOCKER_MY_NETWORK external: true ``` ### - Edit Caddyfile Add global option `acme_dns`
or add `tls` directive to the site-blocks. `Caddyfile` ``` { acme_dns cloudflare {$CLOUDFLARE_API_TOKEN} } a.{$MY_DOMAIN} { reverse_proxy whoami:80 } b.{$MY_DOMAIN} { reverse_proxy nginx:80 tls { dns cloudflare {$CLOUDFLARE_API_TOKEN} } } ``` ### - Wildcard certificate A one certificate to rule all subdomains. But not apex/naked domain, thats separate.
As shown in [the documentation](https://caddyserver.com/docs/caddyfile/patterns#wildcard-certificates), the subdomains must be moved under the wildcard site block and make use of host matching and handles. `Caddyfile` ``` { acme_dns cloudflare {$CLOUDFLARE_API_TOKEN} } {$MY_DOMAIN} { reverse_proxy homer:8080 } *.{$MY_DOMAIN} { @a host a.{$MY_DOMAIN} handle @a { reverse_proxy whoami:80 } @b host b.{$MY_DOMAIN} handle @b { reverse_proxy nginx:80 } handle { abort } } ``` [Here's](https://github.com/caddyserver/caddy/issues/3200) some discussion on this and a simple, elegant way we could have had, without the need to dick with the Caddyfile this much. Just one global line declaration. But the effort went sideways.
So I myself do not even bother with wildcard when the config ends up looking complex and ugly. # Monitoring *work in progress*
*work in progress*
*work in progress* The endgame - [something like this](https://www.reddit.com/r/selfhosted/comments/w8iex6/is_there_a_way_to_see_traffic_statistics_with_the/) googling * https://community.home-assistant.io/t/home-assistant-add-on-promtail/293732 * https://zerokspot.com/weblog/2023/01/25/testing-promtail-pipelines/ * https://github.com/grafana/loki/blob/main/docs/sources/clients/promtail/stages/geoip.md Requires - [Prometheus and Grafana](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/prometheus_grafana) ![caddy_grafana_dashboard](https://i.imgur.com/NmOpGZX.png) ### Metrics Caddy has build in exporter of prometheus metrics, so whats needed: * Edit Caddyfile to [enable metrics.](https://caddyserver.com/docs/metrics) * Edit compose to publish 2019 port.
Likely not be necessary if Caddy and Prometheus are on the same docker network, but its nice to check the metrics at `:2019/metrics` * Edit prometheus.yml to add caddy scraping point * In grafana import [caddy dashboard](https://grafana.com/grafana/dashboards/14280-caddy-exporter/)
or make your own, `caddy_reverse_proxy_upstreams_healthy` shows reverse proxy upstreams, but thats all.
Caddyfile ```php { servers { metrics } admin 0.0.0.0:2019 } a.{$MY_DOMAIN} { reverse_proxy whoami:80 } ```
prometheus.yml ```yml global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'caddy' static_configs: - targets: ['caddy:2019'] ```
But metrics feel kinda not enough, they dont even tell how much which subdomain gets hit.. let alone some access info and IPs. So time for logs and Loki I guess. ### Logs * Have Prometheus, Grafana, Loki working * Create `/var/log/caddy` directory on the docker host * Edit Caddy compose, bind mount `/var/log/caddy` in to caddy container.
Also add Promtail container, that has same bind mount of `/var/log/caddy` directory, along with bind mount of its config file.
Promtail will scrape logs and push them to Loki. * create promtail-config.yml * edit Caddyfile and enable logging at some subdomain
seems global logging might be done by using port 443 as a block, not tested yet * at this points logs should be visible and explorable in grafana
Explore > `{job="caddy_access_log"} |= `` | json` * to-do * *?? edit promtail-config.yml to get desired values ??* * *?? enable somehow geo ip on promtail ??* * *?? make dashboard from logs ??*
docker-compose.yml ```yml services: caddy: image: caddy container_name: caddy hostname: caddy restart: unless-stopped env_file: .env ports: - "80:80" - "443:443" - "443:443/udp" - "2019:2019" volumes: - ./Caddyfile:/etc/caddy/Caddyfile - ./caddy_data:/data - ./caddy_config:/config - /var/log/caddy:/var/log/caddy # LOG AGENT PUSHING LOGS TO LOKI promtail: image: grafana/promtail container_name: promtail hostname: promtail restart: unless-stopped volumes: - ./promtail-config.yml:/etc/promtail-config.yml - /var/log/caddy:/var/log/caddy:ro command: - '-config.file=/etc/promtail-config.yml' networks: default: name: $DOCKER_MY_NETWORK external: true ```
promtail-config.yml ```yml clients: - url: http://loki:3100/loki/api/v1/push scrape_configs: - job_name: caddy static_configs: - targets: - localhost labels: job: caddy_logs __path__: /var//log/caddy/*.log ```
promtail-config-custom-picked-info.yml ```yml clients: - url: http://loki:3100/loki/api/v1/push scrape_configs: - job_name: caddy_access_log static_configs: - targets: # tells promtail to look for the logs on the current machine/host - localhost labels: job: caddy_access_log __path__: /var//log/caddy/*.log pipeline_stages: # Extract all the fields I care about from the # message: - json: expressions: "level": "level" "timestamp": "ts" "duration": "duration" "response_status": "status" "request_path": "request.uri" "request_method": "request.method" "request_host": "request.host" "request_useragent": "request.headers.\"User-Agent\"" "request_remote_ip": "request.remote_ip" # Promote the level into an actual label: - labels: level: # Regenerate the message as all the fields listed # above: - template: # This is a field that doesn't exist yet, so it will be created source: "output" template: | {{toJson (unset (unset (unset . "Entry") "timestamp") "filename")}} - output: source: output # Set the timestamp of the log entry to what's in the # timestamp field. - timestamp: source: "timestamp" format: "Unix" ```
Caddyfile ```php a.{$MY_DOMAIN} { reverse_proxy whoami:80 log { output file /var/log/caddy/a_example_com_access.log } } ```
# Other guides * [gurucomputing caddy guide](https://blog.gurucomputing.com.au/reverse-proxies-with-caddy/) *