And the next amusing diversion, after getting a working startx back, it turns out that the option to set the display size differs between Xephyr and Xvnc
Xephyr -screen 1600x1200
vs
Xvnc -geometry 1600x1200
I was trying to pass -geometry to Xephyr, and it silently accepts it but it doesn't change the size of the display which confused me for a few minutes.