Google open-sources their robots.txt parser and releases an RFC for formalizing the Robots Exclusion Protocol specification

Link information

This data is scraped automatically and may be incorrect.

Title

Google's robots.txt parser is now open source

Word count

289 words