robots.txt works great when everyone is acting in good faith. With a lot of these AI scrapers, I feel like their owners do not operate in good faith. They can't outright ignore robots.txt because...
robots.txt works great when everyone is acting in good faith. With a lot of these AI scrapers, I feel like their owners do not operate in good faith. They can't outright ignore robots.txt because that for sure would result in negative press. So they come up with these tactics where they can act innocent "we are respecting robots.txt, it is just that these are our newer scrapers for our new model.".
You're so right. Honestly, I expect a lot of them not to respect robots.txt whenever they think they can get away with it too. The "go fast and break things"-mentality seems to have gone into...
You're so right.
Honestly, I expect a lot of them not to respect robots.txt whenever they think they can get away with it too.
The "go fast and break things"-mentality seems to have gone into overdrive with AI-companies.
robots.txt works great when everyone is acting in good faith. With a lot of these AI scrapers, I feel like their owners do not operate in good faith. They can't outright ignore robots.txt because that for sure would result in negative press. So they come up with these tactics where they can act innocent "we are respecting robots.txt, it is just that these are our newer scrapers for our new model.".
You're so right.
Honestly, I expect a lot of them not to respect robots.txt whenever they think they can get away with it too.
The "go fast and break things"-mentality seems to have gone into overdrive with AI-companies.