Robots.txt governed the behavior of web crawlers for over thirty years; AI vendors are ignoring it or proliferating too fast to block

patience_limited
2024-02-18T00:56:08Z

Link URL: https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders

Comments URL: https://tildes.net/~tech/1ebl/robots_txt_governed_the_behavior_of_web_crawlers_for_over_thirty_years_ai_vendors_are_ignoring_it

Votes: 41

Comments: 6