Gadgetopia

Enabling historical revisionism: Very interesting note from the Democratic National Committee:

Sometime between April 2003 and October 2003, someone at the White House added virtually all of the directories with ‘Iraq’ in them to its robots.txt file, meaning that search engines would no longer list those pages in results or archive them.

The offending robots.txt file is still out there for your viewing pleasure. The quote above isn’t quite correct — pages already in a search index are not affected by a change in a robots.txt file until the spider goes back to reindex them, at which time they should be removed from the index.