X-forwarding sends so much raw data it's not suitable for over the internet (until fiber and low latency... hello!) but we used to get around this with things like NX (No Machine) which is a layer in the middle that handles compression and optimization.

https://www.nomachine.com