Just a followup on the storage speed test I mentioned above: I decided to do that because I felt like my build jobs on my Forgejo runner were way too slow to my taste, so I started poking around what could it be. In the end, I realized that the runner container is running out of /var/lib/docker, which is stored in the SD card, so there's my bottleneck.
Now I need to find time and patience to move the Docker directory into an NFS share, and see if that speeds up the process. (Forgejo's data directory itself already runs out of NFS).