Spent a better part of today, but I've got dmabuf modifier negotiation fully working in my pipewire screencasting code in niri! This happens to finally fix screencasting on NVIDIA GPUs. Still haven't got any GStreamer pipeline working though, maybe needs a pipewire update.
Apparently, my AMD selects a BGRA format modifier that has two planes. Some out of band info I suppose. And on Intel a preferred BGRA modifier has three entire planes!