logo
Published on Blog Référencement Publicityweb (http://blog.publicityweb.com/blog)

SourceForge blocking Yahoo crawler

By Chris
Created 2006-01-23 13:20

Sometimes, people seems to really want to be excluded from search engine result pages (SERP).

Jeremy Zawodny [1] complained [2] about SourceForge [3] : "You're blocking Yahoo! Search crawlers from your site and have been for several days now." He ask them to unblock the spiders, or to use the robots exclusion standard [4].

The funny part is that Yahoo.com dont provide a robots.txt [5].


Source URL:
http://blog.publicityweb.com/blog/node/160