Embed Notice

@tdp_org This is interesting information and certainly something I will consider making use of myself.

All legitimate crawlers should make it easy to verify the authenticity of the requests they make. I know that for Googlebot the recommended way to verify the authenticity of the request is as follows:

• Do a reverse lookup of the IP address.
• Verify that the resulting name is under the googlebot.com domain.
• Do a forward lookup of the name to verify you get the original IP address back.

I don’t know if similar steps have been published for other legitimate crawlers.