Websites are blocking the wrong AI scrapers (because AI companies keep making new ones) ~tech internet Article 1256 words 18 votes
Robots.txt governed the behavior of web crawlers for over thirty years; AI vendors are ignoring it or proliferating too fast to block ~tech internet Article 3069 words, published Feb 14 2024 41 votes
Google open-sources their robots.txt parser and releases an RFC for formalizing the Robots Exclusion Protocol specification ~comp open source Article 289 words 10 votes