I imported a dump of the database locally (14) and didn't reproduce the errors. So I figured it's either the index or the postgres version. So I tried reindexing on prod and that didn't fix it. So I upgraded Postgres and sure enough it worked.

On 12 it was doing some crazy bitmap heap scan. Now on 14 it's using the index as expected. I assume they improved the query planner... or maybe upgrading just kicked my db in the right way.