How To Verify The GoogleBot Crawling Your Site Is Not A Fake
March 10th, 2007 | 1,988 Views RSS Feed
The Googlebot that just visited your website might not have been from Google. There are rogue bots out there which imitate the Googlebot to steal content. Barry Schwartz has compiled few resources which elaborate upon ways to verify the authenticity of a bot.
Included in the post are some past posts on the issue, a post by Matt Cutts and discussions at Cre8asite Forums. Matt Cutts has explained how to verify the bot with reverse DNS.
Telling webmasters to use DNS to verify on a case-by-case basis seems like the best way to go. I think the recommended technique would be to do a reverse DNS lookup, verify that the name is in the googlebot.com domain, and then do a corresponding forward DNS->IP lookup using that googlebot.com name; eg:
> host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
> host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
I don't think just doing a reverse DNS lookup is sufficient, because a spoofer could set up reverse DNS to point to crawl-a-b-c-d.googlebot.com.
Barry also advises to use CrawlWall to automate the process.
Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!
- del.icio.us
- Digg
- Furl
- Rojo
- StumbleUpon
- Technorati
- Yahoo!
Did you like this article?
Related Posts
Comments
Leave a Reply
SEO Tools
FEATURED CATEGORIES
- adCenter (75)
- AdSense (108)
- AdWords (271)
- Analytics (45)
- AOL (5)
- Ask (98)
- Bing (11)
- Blogging (19)
- Copywriting (1)
- Directory (6)
- Google (1761)
- Industry News (766)
- Keyword Research & Targeting (18)
- Link Popularity (60)
- Live (78)
- Local SEO (7)
- Microsoft (126)
- Mobile Search (13)
- MSN (169)
- PageTraffic Happenings (5)
- Panama (21)
- Pay Per Click (33)
- Search Engine Conferences (153)
- Search Engines (93)
- SEO (215)
- SEO Tools (40)
- Social Media (18)
- Tips & Tricks (12)
- Web Marketing (4)
- Yahoo! (544)
- Yahoo! Search Marketing (64)





