Ok I wasn’t sure at first, but it seems like the performance issue I’m facing is partially due to ref counting. It’s not free!
I’m working in a system I didn’t design, that relies heavily on reference types and inheritance. Given the chance, I probably would have designed it differently, leaning more on value types.