Did Yahoo! Block Google From Crawling Delicious Bookmarks?
February 19th, 2008 | 5,992 Views RSS Feed
During the weekend, Yahoo!'s social bookmarking property del.icio.us seems to have prevented robots of other search engines including Google from spidering the website or fishing out fresh web pages, bookmarks and websites, observes Collin.
It was clear that it was not a simple robots.txt exclusion, rather it was a 404 response that was being shown based on who the requesting User-Agent was.
An observation was made that when the User-Agent was set to Googlebot the 404 error showed up each time, and the same response was received when any page other than the homepage was attempted to be navigated.
"I took a look at del.icio.us' robots.txt," says Collin "and found that it was disallowing Googlebot, Slurp, Teoma, and msnbot for the following:
- Disallow: /inbox
- Disallow: /subscriptions
- Disallow: /network
- Disallow: /search
- Disallow: /post
- Disallow: /login
- Disallow: /rss"
Why on earth Yahoo! may have done this, competition or what. As of now Yahoo has added delicious in its search pages, and that delicious is an important property for Yahoo to bank upon. Apparently, it has begun to make use of its powers from blocking other search engines from taking advantage from spidering the information contained therein.
Moreover, Yahoo! also has an edge over Ask, Google and MSN and that is, it's able to notice the web pages, etc. through human bookmarking, tagging etc. on Delicious, even before other search engine can. Consequently, it leads to better search results and rankings.
In this case Yahoo seems to have made use of its discretionary powers to combat competition. While preventing others from examining your business secrets is a common strategy used by business, it is quite a bold move from the company. What are your thought about it?
There's a discussion about it at Spinn
Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!
- del.icio.us
- Digg
- Furl
- Rojo
- StumbleUpon
- Technorati
- Yahoo!
Did you like this article?
Related Posts
Comments
5 Responses to “Did Yahoo! Block Google From Crawling Delicious Bookmarks?”
Leave a Reply
SEO Tools
FEATURED CATEGORIES
- adCenter (64)
- AdSense (80)
- AdWords (213)
- Analytics (28)
- AOL (3)
- Ask (90)
- Blogging (19)
- Directory (6)
- Google (1477)
- Industry News (652)
- Keyword Research & Targeting (11)
- Link Popularity (50)
- Live (46)
- Microsoft (111)
- Mobile Search (13)
- MSN (169)
- PageTraffic Happenings (4)
- Panama (21)
- Pay Per Click (32)
- Search Engine Conferences (127)
- Search Engines (82)
- SEO (190)
- SEO Tools (33)
- Social Media (18)
- Tips & Tricks (11)
- Web Marketing (3)
- Yahoo! (496)
- Yahoo! Search Marketing (55)
Subscribe To Our SEO Blog
The Blog
The Associates
February 19th, 2008 at 3:31 pm
Hi There Navneet,
I just wanted to add two cents of something i noticed over the weekend as well. I did some searching on Yahoo on Sunday (over the same time period) and noticed that Yahoo was inserting the number of delicious "bookmarks" into their search results. For example, when i did a search for "business blogs"…i noticed that some listings actually had the number of bookmarks in delicious by their listing. So o think its fair to say that they have been experimenting in feeding delicious bookmarks into their search results to offer more comprehensive search results….much like Google's universal search. However i noted that our site "iBlogbusiness.com" (which has several delicious bookmarks) did not have the little icon by the search results. So im not sure how they are determining who they show the bookmark icon to at the moment, but it will be interesting to see what yahoo decides to roll out:)
Best Regards,
Christian
February 19th, 2008 at 5:14 pm
Who cares if delicious will block google or not?
In case you never noticed, all Bookmarks added to delicious have rel="nofollow" set in them,
meaning Google wont/cant/shouldn't use delicous bookmarks to "discover" new sites to index.
February 19th, 2008 at 6:06 pm
The directories being blocked are not important for search and should not get spidered by Google.
E.g. Google does not need to spider the "post a new url" page.
The actual link pages for all users are still spiderable. Of course, as Steve notes, they include rel="nofollow", and so are of little use to Google.
February 20th, 2008 at 9:04 am
As a seocontest2008 participant I am very interested about the social bookmarking issues.
Your news about Yahoo's new move about del.icio.us are really intriguing. Thank you very much for such valuable information. I can't do much about Yahoo's politics, I only have the power of choice to use or not to use del.icio.us services. For the present seocontest2008 purpose I am interested to gain good position on Google, and not on other search engines. If Yahoo has begun to make use of its powers from blocking other search engines from taking advantage from spidering the information contained therein, then I probably will not be interested to use Yahoo's social bookmarking services, hence there are so many other options available.
February 22nd, 2008 at 3:51 am
We've put up a post on our site just to explain why this is actually wrong:
http://www.accuracast.com/search-daily-news/seo-7471/false-reports-about-yahoo-blocking-googlebot-on-delicious/