The Google webmaster help center has been updated recently. There are two new sections which may prove to be particularly useful are:

Using a robots.txt file

New topics have been added in the How Google crawls my site section. These topics include information on:
How to create a robots.txt file
Descriptions of each user-agent that Google uses
How to use pattern matching
How often we recrawl your robots.txt file (around once a day)

Understanding HTTP status codes

This section explains HTTP status codes that your server might return when we request a page of your site. We display HTTP status codes in several places in Google Sitemaps (such as on the robots.txt analysis page and on the crawl errors page) and some site owners have asked us to provide more information about what these mean.

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!