10 votes

Google open-sources their robots.txt parser and releases an RFC for formalizing the Robots Exclusion Protocol specification

0 comments