![[ t ]](/images/structural/icon_twitter.gif){kind=icon}
Sep
9
FBI Robots.txt
Cryptome points us to this link: the robots.txt file at the FBI’s site. I’ll save you the click:
User-agent: *
Disallow: /
This isn’t the first time the government has shown some nervousness about their Web site content: The White House Vs. Search Spiders.
What This Links To
The White House vs. Search Spiders
Enabling historical revisionism: Very interesting note from the Democratic National Committee: Sometime between April 2003 and October 2003, someone at the White House added virtually all of the directories with 'Iraq' in them to its robots.txt file, meaning that search engines would no longer list those pages in results or archive…
Comments are Closed
Thanks to all who participated.
- Why Ad Blocking Kinda Sucks by Deane, 3 hours, 33 minutes ago
- Why Ad Blocking Kinda Sucks by Dave, 7 hours, 25 minutes ago
- Nation Shudders At Large Block Of Uninterrupted Text by Anonymous, 8 hours, 56 minutes ago
- Nation Shudders At Large Block Of Uninterrupted Text by Anonymous, 8 hours, 57 minutes ago
- Joel Spolsky on Twitter by Mark, 12 hours, 30 minutes ago
- Show 5 more »
- The Principle of Least Astonishment by Deane in 2009
- SkySails Maiden Voyage a Success by Deane in 2008
- Sony Computers Sans Crapware by Deane in 2008
- Windows Vista Delayed by Deane in 2006
- Wikipedia Bounties by Deane in 2006
- Show 7 more »