In a post called 'Validate your robots.txt – Googlebot becomes smarter' Sebastian writes that "forgotten –and, until recently, ignored– statements in your robots.txt might change the crawler’s behavior all of a sudden, without notice".

In this regard, John Müller from Google Zürich answers:

"At the moment we will usually accept the "noindex" directive in the robots.txt, but we are not yet at a point where we are willing to set it into stone and announce full support. … I just want to remind everyone again that this is something that may still change over time. Be careful when playing with things like this."

Like all things important, it's not of it's happening, it's when it'll happen that matters.

Click here to subscribe to our RSS feed to get a daily digest of news around search engine industry. PageTraffic SEO Blog is updated four times a day and is ranked as one of the best search engine resources blog by Pandia!