Robots.txt governed the behavior of web crawlers for over thirty years; AI vendors are ignoring it or proliferating too fast to block internet Article 3069 words, published Feb 14 2024 41 votes