With more modern terminals, one can actually get pixel-based GUI full kvm:
https://jexer.sourceforge.io/evolution.html#year2021
(I once had thoughts of using terminals in a data diode type application, precisely because everything goes in-band, could run over even serial port, be saved to file/played back anywhere, etc.)