@tdp_org This is interesting information and certainly something I will consider making use of myself.
All legitimate crawlers should make it easy to verify the authenticity of the requests they make. I know that for Googlebot the recommended way to verify the authenticity of the request is as follows:
- Do a reverse lookup of the IP address.
- Verify that the resulting name is under the googlebot.com domain.
- Do a forward lookup of the name to verify you get the original IP address back.
I don’t know if similar steps have been published for other legitimate crawlers.