@jmorris Oh, #4-40 is only .155mm smaller diameter than M3, so maybe my good experience with M3 carries over...
I do almost always use a tap follower or some other effective guide. I think the spiral flute taps are not at all forgiving of misalignment. But I'm usually doing it on mill or lathe where I have a good normal reference.
I did also make 3D printed jigs for tapping openbuilds v-slot with M5, where I can't typically use either mill or lathe to get that normal reference.