Don't know how to say this without sounding a bit like a terrible person, but I really recommend keeping a small stash of all purpose gifts. These are gifts that no one would be mad to receive, but don't need a specific recipient: nice hand-made soap, hand-made notebooks, keychain bottle-opener/screwdriver, etc.
I just got invited to a white elephant gift exchange and pulled out a "learn to sashiko!" kit I had stashed a couple years ago.