Gadgetopia

80legs sets its web crawler free : 80Legs is offering its crawling engine usage for free.  I’ve used the service, and it’s very impressive.  I crawled a site with about 9,000 pages for something like $0.025.

Why the change? In an email announcing the free service, 80legs says it discovered that many users were crawling less than 100,000 pages. Since the Houston company was charging $2 per 1 million pages crawled, plus $0.03 for every CPU hour used, it wasn’t going to make much money from those users. And since they were researchers, students, and bootstrapped startups, they didn’t have much money to spend anyway.