@cuchaz Cavern has a problem that's very similar; my solution is for the friends to have to brute-force the list until they find something they can decrypt, but not have to do this *often*.
Not sure if this is appropriate in your use-case, but consider whether there's a way to "stabilize" the numbers you're sending your friends; if they can act optimistically on the presumption of the number not having changed since last time (if that's even a meaningful concept) then they get to skip the expensive step whenever they're right.
Barring that, it feels like this might require some fancy math involving combining multiple people's public keys. :-/