Google open-sources their robots.txt parser and releases an RFC for formalizing the Robots Exclusion Protocol specification open source Article 289 words 10 votes