Websites are blocking the wrong AI scrapers (Because AI companies keep making new ones)
18
votes
Showing only topics with the tag "web crawlers".
Back to normal view
-
-
Robots.txt governed the behavior of web crawlers for over thirty years; AI vendors are ignoring it or proliferating too fast to block
41 votes -
The creators of TikTok caused my website to shut down
12 votes -
Spiders
Is anyone here familiar with crawling the web? I’m interested in broad crawling, rather than focusing on particular sites. I’d appreciate pretty much any information about how this is usually...
Is anyone here familiar with crawling the web? I’m interested in broad crawling, rather than focusing on particular sites. I’d appreciate pretty much any information about how this is usually done, and things to watch out for if attempting it.
10 votes -
An Analysis of Cloudflare's Email Address Obfuscation
5 votes