Today I've been visited by kchibisov (Alacritty maintainer) and we've spent several hours benchmarking terminals and editors. 😴
For this test we measured a complex drawing test from vtebench[1]. Key press fills the screen with a complex pattern. I measure the latency from the key press to seeing the pattern at the end of the screen.
Foot ended up firmly ahead, followed by Kitty and Alacritty. Other terminals struggle a bit more with it.
Now for something different: emulators! Here "New Highscore" is the work-in-progress Highscore rewrite @alice is working on, "Old Highscore" is the current latest Highscore git commit, and "GNOME Games" is the latest Games from Flathub.
It's quite interesting how RetroArch seems to have a two-frame spread rather than one, something's off in its processing. Also interesting how MGBA is one frame slower than Gambatte. For Highscore, good to see GTK 4 improving the latency.
As expected, VTE is *really slow* on big window sizes on Wayland due to Cairo and weird repaint timing logic. But what is Black Box doing to lose more than a refresh cycle?
Glad to see Alacritty still on top, but apparently Foot is a tiny bit faster on this test. Kitty loses one refresh cycle for some reason.
This time I wanted to do some more thorough looking at the data before deciding on the thresholding approach, but it seems that the plotly frontend starts to really struggle when you feed it several seconds of data sampled multiple times per millisecond 🙃
Moar measurements: compositors. Since for this test the key presses are slow and there's no continuous redrawing, this should boil down to the amount of work a compositor does on screen update.
Un-vsynced X11 is obviously the fastest; thankfully work to add tearing flips to kernel and Wayland is ongoing.
Surprised to see GNOME Shell be a bit slower than raw Mutter, especially in fullscreen, since it doesn't really do much extra there. Extra surprised GNOME X11 is faster; might be noise.
Next, a more interesting test: editors. For terminal editors I used Alacritty, and I've also added the fast and the slow baselines from the previous tests.
Here Neovim and Helix in text mode are the fastest, followed by nano, which has more spread for some reason. Next we have G-T-E and Builder with quite a bit of spread (@hergertme, any idea what's going on here?), then Helix and Neovim with IDE functionality, and finally VSCode.
2 years ago VSCode was better; maybe my extension setup changed
@whynothugo@YaLTeR huh, that explains it.. always wondered why vte terminals always felt so muddy to type in, and why foot feels like i'm directly pressing the cpu wires with my fingers. nice benchmarks :D
@lanodan it's a tech demo / kitchen sink compositor from Smithay: https://github.com/Smithay/smithay It's by no means fully fledged or optimized, so don't take this as a measure of Smithay's potential performance, but I just wanted to check it out of curiosity.