@quodvideo Hardware acceleration of drawing primitives (2D and 3D) was another area where the X server lagged (and was slow). Everyone wanted direct to hardware OpenGL if they were doing OpenGL at all.
Another driver was fonts and proper text rendering, which basically has to be client side and shipped as bitmaps.
Once you're doing client side rendering, the tree of subwindows approach is relatively terrible (or basically off the table).