Yay, video issues on the Pi 500 are gone!
During the update from Debian 12 to 13, something went wrong with the X11 and Wayland configuration, so I was running a weird "hybrid".
Selecting wayland again in raspi-config and rebooting works, now OpenGL performance is as expected!