Conversation

Notices

1. Embed this notice
   Peter Wemm (karinjiri@soc.crashed.org)'s status on Saturday, 15-Feb-2025 04:35:30 JST Peter Wemm
   AMD PCI passthrough on #FreeBSD for #bhyve is being a PITA with SR-IOV ethernet VFs. Wondering aloud: has anyone here actually seen this work?
   
   This facility is disabled by default and you have to turn it on during early boot. I have done this. Of note, it has been disabled since it was committed ~7 years ago. This makes me suspicious.
   
   While writing this (Rubber Duck debugging FTW) I just realized that I missed a problem: the host is on a vlan tagged port and the switch will drop untagged packets; and the VF wasn't set up for a vlan. Argh!
   
   However, while I chase that down, it would be really useful to hear that somebody is actually using the AMD amdvi PCI passthrough on FreeBSD. Or has seen it work recently.
   
   I'm in this rabbit hole because my initial setup with bhyve/tap/bridge/etc has trouble with the guests doing ethernet mac shenanigans. The guests are swapping mac addresses between themselves (eg: kube-vip, metallb, etc) and by the time this comes out the other side of the if_bridge, packets have the wrong mac on them.
   
   I could just(tm) switch the host to proxmox, but where would the fun be in that? I'm way too stubborn to go there yet.
   • Embed this notice
     Peter Wemm (karinjiri@soc.crashed.org)'s status on Saturday, 15-Feb-2025 04:37:19 JST Peter Wemm

     in reply to
     Well.. I'm a little astonished but after modifying vm-bhyve to write the bare minimum vlan stanzas to the cloud-init network-config it generates - the basic functionality seems to be working. I do have a guest side network-online timeout to resolve.
     
     I did have to have iovctl turn off all the mac filtering/anti-spoofing/etc for ixl0.
     
     I am accumulating tweaks to vm-bhyve for adding functionality. I need more. So far, I need to be able to set an explicit mac for a VM - to match the mac assigned to the VF. And be able to specify pre-built cloud-init config files.
     
     The cidata seed.iso is definitely better as a virtio-blk rather than ahci-cd because ahci-cd adds about one extra second of boot time for linux guest kernels. I forsee some patches heading upstream soon.
     
     Anyway, to answer my first question: yes, AMD pci passthrough does seem to work - when enabled.
     feld likes this.