@whitequark Better than the debug hell I went through in my thesis work.
I was using a gdbserver I wrote to debug firmware I wrote, compiled against a libc I wrote, running on a CPU I wrote, hooked up to an interconnect I wrote, on a board I designed, using a FTDI-based debug dongle I designed.
And something was going wrong somewhere in the stack and the firmware was locking up (not a full deadlock, instructions were still executing, but it was stuck in a polling loop or something and not making any forward progress). I trusted nothing and had no idea where in the stack things were going wrong.
And it was on a very full xc6slx25 that had enough spare gates to fit *either* JTAG debug support on the CPU *or* a homebrew ILA to sniff a few internal signals. But not both.