The reason for the large negative step errors on first sync, if anyone is curious, is that the RTC is clocked by the HSE oscillator (25 MHz source) which does not keep running across resets or power-down.
Or, more importantly, when flashing a new firmware via JTAG.
So any time I firmware update the board the RTC stops for a few seconds and it lags behind actual time until the next NTP sync.
(I wasn't originally planning on using the RTC on this board so I never put a low-speed crystal on it)