Blog post: An illustration of how much X cares about memory usage https://utcc.utoronto.ca/~cks/space/blog/unix/XServerBackingStoreOptional
tl;dr: if another window covers up part or all of your window, the X protocol allows the server to throw away those parts of your window, stop you drawing new things to them, and then require you to repaint them later, all so that the server doesn't have to allocate an off-screen buffer for your pixels.
(X clients used to render 'directly' to the screen, not to an intermediate buffer.)