On the subject of 'nobody really understands how SD works', here's a recent paper that says that every SD model has been trained wrong, which reduces it's capability to produces images that differ from average brightness:

https://arxiv.org/pdf/2305.08891.pdf

This is the flaw that something like noise offset tries to correct, but this paper suggest a real fix.