Got sub-rate modes working, and configurable via SCPI (but not yet in the scopehal driver).
But there's an interesting reflection visible in the eye in sub-rate mode.
I currently have two suspects.
1) The Mini-Circuits BLK-18-S+ SMA DC blocks I have on the signal path, but mated to a 2.92mm connector. It's about the right length to be causing the issue, and is deliberately not torqued down because of the mismatched connector types. I have a pair of BLK-K44+ 2.92mm DC blocks inbound that I wanted to have anyway. If the issue goes away tomorrow when they get here I'll know.
2) Differential via on the PCB without a return current via near it (board design bug that slipped through design review, I have return vias on every high speed signal via except this pair and the one next to it).