Ah, I figured out why all this is happening. Not only why everything looks incredibly bad but also why it literally hurts my nose to wear.
I have long hair.
The PSVR2 is designed to kinda grip your skull from under the back in order to stay in one place. But the front is so heavy compared to the back that, on my head, it just slips forward until the whole weight of the headset is resting on my nose bone.
That puts it at the wrong angle and makes everything look much worse, prevents you from keeping it still, etc.