Glad to report that I've figured out what was causing some activities to spike to up to 1.4 seconds and fixed it.
When an Activity arrives if it's the first time one from that instance or that user has arrived, there are a few extra network requests PieFed needs to make to ensure the Activity data is fully fleshed out. Fortunately most of them can be deferred to future background tasks.