Conversation

Notices

1. Embed this notice
   Dash 🇸🇬 (dashrandom@kopiti.am)'s status on Wednesday, 10-Apr-2024 16:33:58 JST Dash 🇸🇬

   After much wrangling with Traefik plugins configs, and some good old trial and error, finally managed to find a way to restore original visitor IPs to my docker containers on a Cloudflare proxy + Traefik reverse proxy deployed in a docker container on a bridged network set up.

   • Embed this notice
     Dash 🇸🇬 (dashrandom@kopiti.am)'s status on Wednesday, 10-Apr-2024 16:34:36 JST Dash 🇸🇬

     in reply to

     The fact that Traefik in a docker container doesn't pass on client X-forwarded-for or X-Real-IP headers for reverse proxied requests, but instead substitutes them with docker bridged network IPs, is a bit unintuitive especially when you consider that HTTP Requests are part of the TCP protocol, which runs at Layer 4 of the OSI model and not Layer 3. Wait but Traefik is an application? Doesn't it run on Layer 7?

   • Embed this notice
     Dash 🇸🇬 (dashrandom@kopiti.am)'s status on Wednesday, 10-Apr-2024 16:35:27 JST Dash 🇸🇬

     in reply to

     To make things muddier, reverse proxies in particular complicate the entire thing as they perform routing functions as well. Traefik in particular functions as a virtual router in specific instances. Throw in the fact that docker itself has networking capabilities that are either host or bridge dependent and you get the perfect recipe for a stew of wtf is going on here.

   • Embed this notice
     Dash 🇸🇬 (dashrandom@kopiti.am)'s status on Wednesday, 10-Apr-2024 16:38:21 JST Dash 🇸🇬

     in reply to

     So it turns out that there are a few problems to solve.

     1/ Cloudflare send X-Real-IP in a CF-Connecting-IP HTTP Header instead of X-Forwarded-For or X-Real-IP because Cloudflare *itself* acts as a proxy. More here: https://developers.cloudflare.com/support/troubleshooting/restoring-visitor-ips/restoring-original-visitor-ips/
     2/ Traefik is a reverse proxy, it takes in requests and forwards them. However, when you deploy Traefik in a docker container and assign it a bridge network, it takes in traffic through the docker NAT, not the host network.

   • Embed this notice
     Dash 🇸🇬 (dashrandom@kopiti.am)'s status on Wednesday, 10-Apr-2024 16:42:56 JST Dash 🇸🇬

     in reply to

     3/ Traefik also forwards requests, and again, in a docker container on a bridged network, it forwards the requests to containers through the docker NAT (which makes perfect sense)

     So the solution required resolving each of these problems, first by exposing the ports Traefik connects to on the host interface. More info here: https://docs.docker.com/compose/compose-file/05-services/#long-syntax-3

   • Embed this notice
     Dash 🇸🇬 (dashrandom@kopiti.am)'s status on Wednesday, 10-Apr-2024 16:50:08 JST Dash 🇸🇬

     in reply to

     Next, by allowing all forwarded headers into our Traefik entrypoints to be forwarded to the containers by modifying the static Traefik config. (Option of being lazy and going insecure: true or defining the cloudflare IP CIDRs one by one)

     Lastly, rewriting the received HTTP Header that Traefik receives with a middleware plugin (shoutout to github.com/Paxxs/traefik-get-real-ip) so that CF-Connecting-IP is now forwarded as X-Forwarded-For or X-Real-IP when the containers receive it.

   • Embed this notice
     Dash 🇸🇬 (dashrandom@kopiti.am)'s status on Thursday, 11-Apr-2024 10:14:40 JST Dash 🇸🇬

     in reply to

     tl;dr: Don't use Traefik unless you know exactly what the hell it does. Addtionally, don't use Traefik in a docker container unless you are intimately familiar with networking.

     Afterthought: idk how the hell the average web developer who just builds apps is going to figure this out. This requires some understanding about networking + containerization tech. Just looking at the number of posts on the Traefik forums asking the same question is quite telling.

   • Embed this notice
     Dash 🇸🇬 (dashrandom@kopiti.am)'s status on Thursday, 11-Apr-2024 20:49:25 JST Dash 🇸🇬

     • houfu

     @houfu nginx proxy manager is a lot easier for most things.

     I wouldn't say Traefik is necessarily bad but it is complicated. Same for docker. The issue is because they hide most of their complexity from the average user with some templated deployments, it becomes hard to untangle once you need to touch those complicated parts to achieve stuff.

     That being said I'm not going back to native installs and equally complex networking. What we have now is simpler and more complex at the same time.

   • Embed this notice
     Dash 🇸🇬 (dashrandom@kopiti.am)'s status on Friday, 12-Apr-2024 19:30:17 JST Dash 🇸🇬

     • houfu

     @houfu Fun fact, I just learnt today that my problem is behaviour specific to the docker container orchestration engine. If I was using Kubernetes, I wouldn't get have this issue...

     Might need to migrate 😕