Google open-sources their robots.txt parser and releases an RFC for formalizing the Robots Exclusion Protocol specification

Deimos
2019-07-01T19:39:22Z

Link URL: https://webmasters.googleblog.com/2019/07/repp-oss.html

Comments URL: https://tildes.net/~comp/f3v/google_open_sources_their_robots_txt_parser_and_releases_an_rfc_for_formalizing_the_robots

Votes: 10

Comments: 0