Members at Webmaster World discusses about the problem of server header status codes. It is one of those topics that aren't touched often!

According to the Webmasters World thread, almost every server can return an incorrect HTTP status in the server header. But, Microsoft IIS (Internet Information Service) has a particularly long history of such errors been shown. For example:

Tried to set up a "custom 404" page, so your users get a friendly human like message.

When a bad URL is requested, the server responds with a 302 redirect to the custom error page (often called 404.html for example). Then the custom message is returned with a 200 OK http status.

The user agent is never informed by the server that the status of the requested URL should be 404. As a result a '302 temporary redirect' is issued. Unless the server informs the crawler about the correct status of requested URL, it won't matter what error code is written in the web page. Crawlers (including Googlebot) communicate with servers and not individual web pages.

Some of the interesting threads at Webmasters World:

"Any bad URL can be indexed according to the standard handling of an internal 302 redirect:

a) The content of the redirect's target URL is indexed.

b) With the original URL as the location.

So the bad URLs can start piling up as duplicate URLs for the same exact content."

Microsoft's .NET platform and IIS are particularly susceptible to this kind of errors. Inspite of the fact that the default error handling returns a correct 404 status code, still it is the way by which custom error messages are set up that pose the problem.

"For the IIS user, there is one other caution I should mention about 404 handling. If you are using .NET, then there are two levels of error handling: at the IIS level and at the .NET level. It is also common to find that only one of these two levels is set up correctly. So when you're stress testing your URLs, try a bad URL with an .asp (.aspx) extension, and also try a bad url with a .htm extension.

I felt it was important to focus on this issue in a dedicated thread - it's coming up much too often, both in threads here and in the sites of new clients that I evaluate.

The problem is by no means confined to IIS, either. For example, I see it on Apache servers running Tomcat for .jsp pages."