Thanks @ryan for complaining.
I did some digging and discovered a YEARS old misconfiguration which caused *some* jobs to sporadically timeout and fail.
Nerds: NAS uses two redis caches, a main persistent cache written to disk and a secondary LRU cache that only exists in memory.
Bug: I had a duplicate /etc/hosts entry naming both servers as "redis" on the newest server.
It may not always be DNS, but it's always name resolution.
Issue resolved.