Conversation

Notices

1. Embed this notice
   feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 05:32:34 JST feld
   Is Linux networking fucking drunk or what?
   
   So I have a VPS with an additional static ip. I have a wireguard tunnel. I add a static route to pass that static IP through the tunnel.
   
   This part is fine. I can ping the static IP over the tunnel. Everything is fine on the other side, anyway.
   
   But remote traffic / traffic from the internet cannot reach the IP. It's not firewall. The static IP is definitely routed to this Linux server, so it should be able to accept the incoming traffic and forward it (sysctls are correct, that's not the issue).
   
   So I take a shot in the dark. I add the static IP as an additional loopback address. Still with the static route for the same address saying it goes over Wireguard.
   
   And it fucking works??????
   
   How do I have the IP address literally on a loopback *AND* the traffic for that IP is being routed across the Wireguard tunnel correctly?
   
   
   WHAT THE FUCK ARE YOU DOING LINUX
   • Embed this notice
     dilbert 1 (sun@shitposter.world)'s status on Friday, 28-Mar-2025 05:33:20 JST dilbert 1

     in reply to
     @feld it is so bad
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 05:46:12 JST feld

     in reply to

     • dilbert 1
     @sun so I restarted the server to see if it would persist and now it's all broken and not working again. My loopback trick is no longer functional. (it should have never worked in the first place)
     
     This network stack really is a piece of trash.
     snacks and dilbert 1 like this.
   • Embed this notice
     narcolepsy and alcoholism :flag: (hj@shigusegubu.club)'s status on Friday, 28-Mar-2025 05:46:49 JST narcolepsy and alcoholism :flag:

     in reply to

     • dilbert 1
     @sun @feld CW: Very Hot Take, i'm truly really sorry to Feld you have to read this so please don't
     
     still better than FreeBSD's one. I have NAS, back when it had FreeNAS, BSD-based OS, it would regularly shit the bed when it comes to Realtek Ethernet. I had to hunt down ethernet adapter that's "good" and even with it it would shit the bed, just not as often. The most common excuse i've heard is something like "freebsd is so effecient it makes NIC crash". Now that FreeNAS is TrueNAS and it's linux-based instead of freebsd-cringe it literally never crashes.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 05:48:10 JST feld

     in reply to

     • narcolepsy and alcoholism :flag:
     • dilbert 1
     @hj @sun the Realtek NIC is not the network stack, you had a driver problem
     
     nobody should use a Realtek. They're awful.
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: likes this.
   • Embed this notice
     narcolepsy and alcoholism :flag: (hj@shigusegubu.club)'s status on Friday, 28-Mar-2025 05:50:55 JST narcolepsy and alcoholism :flag:

     in reply to

     • dilbert 1
     @feld @sun that's "Your computer cannot run Windows 11" kind of excuse. I think i even reported the driver issue to no avail. I don't give two fucks about shitty drivers for commonly available hardware. It's not like it's some sort of train controller joystick that requires specialized hardware, it's a fucking NIC integrated into MoBo. Shit's everywhere, man.
     Phantasm likes this.
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: repeated this.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 05:55:57 JST feld

     in reply to

     • narcolepsy and alcoholism :flag:
     • dilbert 1
     @hj @sun This is like complaining back in the day that Linux didn't support winmodems very well
     
     It was shit hardware that required you do almost all the work in software. Don't blame the OS devs for not giving two shits about a terrible common piece of hardware
     
     Good NICs:
     
     - Intel, most of them anyway
     - Mellanox/Nvidia
     - Solarflare
     - Chelsio
     
     Bad NICs:
     
     - Realtek
     - Broadcom
     
     
     The good NICs even have developers from the respective companies writing the drivers for them on FreeBSD. I don't believe Broadcom or Realtek has any devs contributing anything at all.
     Doughnut Lollipop 【記録係】:blobfoxgooglymlem: likes this.
   • Embed this notice
     narcolepsy and alcoholism :flag: (hj@shigusegubu.club)'s status on Friday, 28-Mar-2025 05:59:39 JST narcolepsy and alcoholism :flag:

     in reply to

     • dilbert 1
     @feld @sun just for the record i also just as bitter towards Linux and Whorecom (get it? Broad com!). I don't care if you have to do stuff in software - i do things in frontend that should've been done in backend.
     
     >Broadcom or Realtek has any devs contributing anything at all.
     
     Broadcom BARELY works on linux. Realtek works just fine. Intel is all over the place, even their CPUs have mediocre support if you ask me (i'm AMD all the way)
   • Embed this notice
     Dewoo Alt-dog (dwaltiz@pleroma.soykaf.com)'s status on Friday, 28-Mar-2025 06:00:06 JST Dewoo Alt-dog

     in reply to

     • narcolepsy and alcoholism :flag:
     @feld @hj "nobody should use a Realtek"
     yeah, let's just trash 99% of consumer motherboards
     narcolepsy and alcoholism :flag: and Phantasm like this.
   • Embed this notice
     narcolepsy and alcoholism :flag: (hj@shigusegubu.club)'s status on Friday, 28-Mar-2025 06:00:18 JST narcolepsy and alcoholism :flag:

     in reply to

     • Dewoo Alt-dog
     @dwaltiz @feld "your computer cannot run FreeBSD 11" t. Microsoft
   • Embed this notice
     I am Water (slicerdicer@friedcheese.us)'s status on Friday, 28-Mar-2025 06:04:54 JST I am Water

     in reply to

     • narcolepsy and alcoholism :flag:
     • dilbert 1
     @feld @hj @sun Broadcom is awful.
     narcolepsy and alcoholism :flag: likes this.
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Friday, 28-Mar-2025 06:05:28 JST Phantasm

     in reply to

     @feld What I suspect happens is that Wireguard drops your packets silently (this is by design), if they aren't in the AllowedIPs list for that specific peer.

     You should be able to that happen, if you run this and watch dmesg

     echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control

     I might also be completely misinterpreting your setup.

   • Embed this notice
     I am Water (slicerdicer@friedcheese.us)'s status on Friday, 28-Mar-2025 06:09:59 JST I am Water

     in reply to

     • narcolepsy and alcoholism :flag:
     • Dewoo Alt-dog
     @hj @dwaltiz @feld All my computers except the shit with Broadcom can run FreeBSD. That would be my MacBook Pro that is incapable.
     narcolepsy and alcoholism :flag: likes this.
   • Embed this notice
     I am Water (slicerdicer@friedcheese.us)'s status on Friday, 28-Mar-2025 06:11:55 JST I am Water

     in reply to

     • narcolepsy and alcoholism :flag:
     • dilbert 1
     • I am Water
     @feld @hj @sun >> Choose a shitty DAC -> get shitty audio
     
     And that’s why mars attack happened. They were trying to tell us to fix our dac and we kept thinking they said ack. Simple misunderstanding leads to planetary war.
     feld likes this.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 06:13:34 JST feld

     in reply to

     • Phantasm
     @phnt nope, that part is taken care of actually.
     
     I'm just trying to get a static IP that the VPS is supposed to have on its own interface to actually be routed through Wireguard to a computer of my choosing.
     
     It was temporarily working which I think was a Linux kernel bug because the traffic should have been getting delivered to the loopback interface at the time. Now I can't get it to work again :)
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 06:16:04 JST feld

     in reply to

     • Nico, Harbinger of Memes
     @nico no, that doesn't make sense. Routers route packets all the time without needing the addresses to be on loopbacks.
     
     The VPS provider assigned the IP to my VPS, so their end should have a route going to my primary IP. I should be able to route that IP to any other directly connected network like over the VPN tunnel to a VM/machine that has that static IP on its own interface.
   • Embed this notice
     Nico, Harbinger of Memes (nico@toot.exchange)'s status on Friday, 28-Mar-2025 06:16:05 JST Nico, Harbinger of Memes

     in reply to

     @feld Linux server wasn’t configured as gateway until you added it as loopback.

   • Embed this notice
     I am Water (slicerdicer@friedcheese.us)'s status on Friday, 28-Mar-2025 06:17:07 JST I am Water

     in reply to

     • narcolepsy and alcoholism :flag:
     • dilbert 1
     @hj @feld @sun Whorecom get 4 nic? You can passthru
     
     Nic 0 -> VM 1
     Nic 1 -> VM 2
     Nic 3 and 4 VM 3
     
     Nic 0, 2, 3 work fine. Nic 1 fires and takes down both 0 and 1.
     
     Fuck them.
     
     It’s like going to Olive Garden and they are out of breadsticks.
     narcolepsy and alcoholism :flag: likes this.
   • Embed this notice
     narcolepsy and alcoholism :flag: (hj@shigusegubu.club)'s status on Friday, 28-Mar-2025 06:17:25 JST narcolepsy and alcoholism :flag:

     in reply to

     • dilbert 1
     • I am Water
     @SlicerDicer @feld @sun >It’s like going to Olive Garden and they are out of breadsticks.
     
     oh how i missed this
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Friday, 28-Mar-2025 06:18:36 JST Phantasm

     in reply to

     • narcolepsy and alcoholism :flag:
     • dilbert 1
     @hj @feld @sun
     
     >Intel is all over the place
     There also used to be a big problem with certain older Intel NICs on FreeBSD that used the em driver. They would lockup and effectively drop all incoming packets. That was more than a decade ago in the FreeBSD 8 to FreeBSD 10 era I think.
     narcolepsy and alcoholism :flag: likes this.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 06:19:04 JST feld

     in reply to

     • narcolepsy and alcoholism :flag:
     • Phantasm
     • dilbert 1
     @phnt @hj @sun yeah the em driver sucks, the ix driver is the good one
   • Embed this notice
     narcolepsy and alcoholism :flag: (hj@shigusegubu.club)'s status on Friday, 28-Mar-2025 06:20:34 JST narcolepsy and alcoholism :flag:

     in reply to

     • Phantasm
     • dilbert 1
     @phnt @feld @sun i have two machines, both have ATi GPUs
     
     one has Intel processor (and chipset), things break down after system goes to sleep and wakes up
     
     other has AMD processor (and chipset), things only work well after a sleep and wake up.
     
     
     like fucking poetry
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Friday, 28-Mar-2025 06:24:09 JST Phantasm

     in reply to

     @feld I've never tried doing that with Wireguard, so I can't really help more than this. The last remaining piece of information I have is that wg-quick does some shenanigans with routes and maybe that broke it/prevents it from working even though it shouldn't. For example if you specify AllowedIPs on peer to 0.0.0.0, ::0, it makes that the default route for everything. Which is something you probably already know.

   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 06:25:15 JST feld

     in reply to

     • Nico, Harbinger of Memes
     @nico I'm sorry but this is just wrong and I can prove it
     
     from my desktop:
     
     > ping 15.204.35.21
     PING 15.204.35.21 (15.204.35.21): 56 data bytes
     64 bytes from 15.204.35.21: icmp_seq=0 ttl=63 time=0.266 ms
     
     Where is 15.204.35.21?
     
     It goes to my default gateway / firewall.
     
     But that IP isn't on any interfaces there
     
     Instead there's a route that says:
     
     # netstat -rn | grep 15.204
     15.204.35.21 10.27.2.5 UGH1 bridge4
     
     send that IP address to my FreeBSD jail at 10.27.2.5 (never mind that this route is actually installed on my firewall with iBGP from inside that jail)
     
     and in that jail is where the IP address can be found on an interface
     
     e0b_web: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
     options=8<VLAN_MTU>
     ether 0e:40:b2:db:80:58
     hwaddr 02:82:d7:33:3e:0b
     inet 10.27.2.5 netmask 0xffffff00 broadcast 10.27.2.255
     inet 15.204.35.21 netmask 0xffffffff broadcast 15.204.35.21
     inet6 fe80::c40:b2ff:fedb:8058%e0b_web prefixlen 64 scopeid 0x58
     groups: epair
     media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
     status: active
     nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
     
     
     
     Routing does not work the way you think it does
   • Embed this notice
     Nico, Harbinger of Memes (nico@toot.exchange)'s status on Friday, 28-Mar-2025 06:25:16 JST Nico, Harbinger of Memes

     in reply to

     @feld having the IP on an interface doesn’t mean it routes. In order to accept a packet it must be a)addressed to an IP the device has in its interfaces, or b)be directly routable to a nearby gateway with a static route.

     The server will not accept the packet unless it’s on the loopback, in which case it says “oh, that’s me” and take the packet, and then loopback it and it falls through the WireGuard static route and is sent the way you want.

     1/2

   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Friday, 28-Mar-2025 06:26:30 JST Phantasm

     in reply to

     • narcolepsy and alcoholism :flag:
     • dilbert 1
     @hj @feld @sun I usually have bad experience with both. Sleep on my Intel laptop under Linux breaks every few releases. And on an AMD APU desktop, waking up from sleep crashes the GPU, if you donẗ have the correct kernel command line argument set and it fails to boot usually twice a year after a kernel update (I blame Ubuntu).
     narcolepsy and alcoholism :flag: likes this.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 06:27:13 JST feld

     in reply to

     • Phantasm
     @phnt it works from that wireguard host
     
     $ ping 15.204.35.21
     PING 15.204.35.21 (15.204.35.21) 56(84) bytes of data.
     64 bytes from 15.204.35.21: icmp_seq=1 ttl=63 time=32.7 ms
     
     the route is working, it goes over the tunnel
     
     It's just that traffic from the outside world that hits that host's NIC should be forwarded by the kernel over the wireguard tunnel. It's just not even trying. But it SHOULD. The route is there.
     Phantasm likes this.
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Friday, 28-Mar-2025 06:35:16 JST Phantasm

     in reply to
     @feld Stupid idea ahead: do you have sysctl net.ipv4.ip_forward set to 1?
   • Embed this notice
     Nico, Harbinger of Memes (nico@toot.exchange)'s status on Friday, 28-Mar-2025 06:37:06 JST Nico, Harbinger of Memes

     in reply to

     @feld lets replace your WAN IP for your VPS with a name, “Kyle”. WG endpoint can be Bob.

     You are asking traffic for Kyle to be accepted by Kyle and then forwarded to Kyle over a tunnel between Kyle and Kyle but it’s actually Bob. You are making this 1000% harder by trying to map your IP to a machine that doesn’t have it by IP address.

     You have a few ways to do this:
     1. NAT all incoming traffic to the IP address of the machine on the other end of the tunnel(all Kyle goes to Bob)

     1/2

   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 06:37:06 JST feld

     in reply to

     • Nico, Harbinger of Memes
     @nico buddy, I used to work for Cisco. I worked at an ISP for years. This is just basic shit that I've done countless times except instead of a Wireguard tunnel it was another point to point tunnel like a GIF or a GRE tunnel, sometimes with IPSEC layered on top.
     
     It's. Just. Routing.
     
     If the traffic for a subnet is routed to a router, you can route that same traffic out another interface over another tunnel. You just need the route in your table so it knows where the next hop is, and it will work as long as you don't exhaust the TTL. It's not rocket science. This is how the entire internet works. How do you think you can reach Google if your ISP doesn't own Google's IPs? They have a route saying "send traffic for 8.8.8.8 out this interface, someone on the other end knows where it goes after that"
     dilbert 1 likes this.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 06:38:01 JST feld

     in reply to

     • Phantasm
     @phnt lol yes :) but I was seriously wondering if this was somehow broken too lol
     
     $ sudo sysctl net.ipv4.ip_forward
     net.ipv4.ip_forward = 1
     Phantasm likes this.
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Friday, 28-Mar-2025 06:39:23 JST Phantasm

     in reply to
     @feld Than I'm out of ideas what can be wrong. Masquerading shouldn't matter in this case I think. Sorry.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 06:48:21 JST feld

     in reply to

     • Phantasm
     @phnt yeah I really don't want masquerade or DNAT because then I lose useful logs and stuff as I won't have the source IPs
     
     and I don't want to terminate the connections at a webserver on the VPS either
   • Embed this notice
     Phantasm (phnt@fluffytail.org)'s status on Friday, 28-Mar-2025 06:54:07 JST Phantasm

     in reply to
     @feld I've done something similar with relayd and pf on OpenBSD not so long ago, but the setup for that isn't friendly either. relayd can do transparent proxy which preserves the source IP, but you end up having two connections on the "edge". One is incoming (to be relayed) and the other one is outgoing, which is something not really ideal.
     
     It was also a pain to setup, because the documentation for that is a single mail in the OpenBSD mailing list from many years ago.
     feld likes this.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 06:55:46 JST feld

     in reply to

     • Nico, Harbinger of Memes
     @nico
     
     > but the whole point is the packets to my ISP aren’t addressed to Google, they’re addressed to my ISP
     
     That's wrong, capture the packets.
     
     If your ISP gives you a default gateway of 1.2.3.4 it does not put the destination in the packet as 1.2.3.4
     
     The destination address will say 8.8.8.8 if you're trying to reach 8.8.8.8.
     
     Your router/firewall says: "8.8.8.8? I don't have that, the most specific route that matches is 0.0.0.0/0 which goes out ens2 to the default gateway 1.2.3.4, so I'll just send this packet there. What's 1.2.3.4's MAC address? Ok, rewrite the frame to use that MAC and send it out ens2 now"
     
     The IP address is not changed.
     Phantasm likes this.
   • Embed this notice
     Nico, Harbinger of Memes (nico@toot.exchange)'s status on Friday, 28-Mar-2025 06:55:47 JST Nico, Harbinger of Memes

     in reply to

     @feld but the whole point is the packets to my ISP aren’t addressed to Google, they’re addressed to my ISP. My ISP doesn’t have Googles IP. Packet for 8.8.8.8 goes to the default gateway but the default gateway is not 8.8.8.8.

     You’re asking me to have Googles IP, my ISP to have Googles IP and Google to have their IP and somehow get it from A to B to C in the right order.

     Sorry for bothering you if I’m not being helpful.

   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 08:07:05 JST feld

     in reply to

     • Phantasm
     @phnt oh my god I think I got it working, just gotta reboot to test
     Phantasm likes this.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 08:07:05 JST feld

     in reply to

     • Phantasm
     @phnt yes it's working. Hooray!
     
     The issue I wasn't noticing? Packets were being sent through Wireguard correclty, return traffic was not happening.
     
     Had to go to the other side's Wireguard config and set AllowedIPs to 0.0.0.0/0 because the source IP on the traffic coming through was not going to be constrained to my tunnel addresses. Obviously. Duh.
     
     It's always obvious in hindsight.
     
     
     So now I have an OVH static IP inside my home office thanks to Wireguard. :toot:
     Phantasm and dilbert 1 like this.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 10:47:10 JST feld

     in reply to

     • Phantasm
     • Crispy Branzino
     @Nimbius666 @phnt so the one thing I really needed was on the VPS
     
     sudo sysctl net.ipv4.conf.all.proxy_arp=1
     
     that gets the traffic in
     
     The other problem is that I have to do AllowedIPs=0.0.0.0/0
     
     but this pushes all my traffic out the Wireguard tunnel which breaks internet access because normal traffic is going to have a source IP of my jail, which is RFC1918
     
     Fortunately all the things I really need to access are also on local networks so a more specific static route fixes that. I'll still be able to get packages and stuff, but you can't make new network connections from here out to the internet.
     
     Luckily I don't really need that so it's kind of a security feature anyway as if someone compromised the webserver and got a shell they couldn't reach the real internet from it. But I can still reach my package server, do updates etc because all those operations can happen from outside the jail
     
     Anyway, it works. This wasted a lot of my energy today but goddamn it I wanted a working static IP on this because my ISP won't sell me any
     
     I miss the /27 I used to have. Or the /23 from ages ago.
     Phantasm likes this.
   • Embed this notice
     Crispy Branzino (nimbius666@comp.lain.la)'s status on Friday, 28-Mar-2025 10:47:11 JST Crispy Branzino

     in reply to

     • Phantasm
     @feld @phnt the number of times this hits me is pure comedy gold.
     Phantasm likes this.
   • Embed this notice
     feld (feld@friedcheese.us)'s status on Friday, 28-Mar-2025 11:21:37 JST feld

     in reply to

     • Phantasm
     • Crispy Branzino
     @Nimbius666 @phnt Crap I will need NAT because otherwise it can't do LetsEncrypt / ACME for my public certs. Guess I'll have to add that onto the VPS. Not a big deal tho