The Numbers felt this shift in two distinct waves. The first started around 2024:
We saw a big increase in crawls as AI training joined the search engine crawlers. The AI crawlers are generally less well-behaved than the search engines, which increased the management tasks for us to keep the site running smoothly.
And the second wave was stronger and more damaging:
Around December 2025, we saw another big spike in traffic which I attribute to agentic AI: a combination of AI agents that scrape sites in response to prompts, and people being able to write agents that scrape sites.
Bruce says that only 10% of their traffic is from humans browsing the site, with the rest coming from AI bots and automated traffic.
Around 90% of our time was spent keeping the existing site running while we spent our spare moments working on a new and improved system.
The Internet has always been a hostile place but it's becoming more so. I think we're going to see many websites either going behind a proxy like Cloudflare, or converting into a static website so...
The Internet has always been a hostile place but it's becoming more so. I think we're going to see many websites either going behind a proxy like Cloudflare, or converting into a static website so that the web pages can be served by a CDN.
The numbers here were really eye-opening read for me. Yea I've heard the headlines about dead internet theory, bots > than humans, and the prevalence of crawling with ai, but I really had no idea...
The numbers here were really eye-opening read for me. Yea I've heard the headlines about dead internet theory, bots > than humans, and the prevalence of crawling with ai, but I really had no idea of the scale, so seeing the ratio of 38,000:1 anthropic crawls:visitor was honestly shocking.
Also really interesting to see how, in the case of The Numbers, they're treating the crawlers as the audience to increase paid services.
Flip side, though: this really highlights how irresponsible the ai companies comes across to me, releasing such powerful tools into the wild and only then playing whackamole to put up guardrails and fix problems that could reasonably have been anticipated (though I'll grant, devious people can be clever as hell).
I’ve been saying that ChatGPTs Release without many many more disclaimers was incredibly reckless. The amount of people not realizing that just because it’s written well, it doesn’t mean that it’s...
Flip side, though: this really highlights how irresponsible the ai companies comes across to me, releasing such powerful tools into the wild and only then playing whackamole to put up guardrails and fix problems that could reasonably have been anticipated (though I'll grant, devious people can be clever as hell).
I’ve been saying that ChatGPTs Release without many many more disclaimers was incredibly reckless. The amount of people not realizing that just because it’s written well, it doesn’t mean that it’s correct. Is astounding.
Especially early models. Like I’ve had a friend who’s ChatGPT confirmed his conspiracy theories like the moon landing being fake and others
Ah, of course the underlying reason was prediction markets. I figured there had to be some financial reason why anyone would bother throwing so much traffic at the site. Simply scraping answers is...
Ah, of course the underlying reason was prediction markets. I figured there had to be some financial reason why anyone would bother throwing so much traffic at the site. Simply scraping answers is probably a good share too, but that wouldn't explain the level of dedication to rapid data access.
The only saving grace is private and niche communities that is hard to craw or not designed for SEO. Back in the day, every website was designed to be indexed by Google but with today's tech we...
The only saving grace is private and niche communities that is hard to craw or not designed for SEO. Back in the day, every website was designed to be indexed by Google but with today's tech we can actually do the opposite and make it as hard as possible for AI bots to crawl your site. Cloudflare is already working on this.
Any website with great content will eventually wall up. It just makes sense cuz otherwise you'd be giving it free to OpenAI and Anthropic and basically let them make money while you suffer.
Funny how the internet started to be as open as possible but actually now the best way to build a community is to make it as close and as niche as possible.
Without proper regulation around this stuff, there no other way to operate sanely. Unless you are fine spending yours (or someone else's) full-time efforts fighting bots. Prediction markets are...
Without proper regulation around this stuff, there no other way to operate sanely. Unless you are fine spending yours (or someone else's) full-time efforts fighting bots.
Prediction markets are 100% gooig to come under fire in the coming few years, so I guess in this specific instance that will bring relief here.
Oh my comment is a lot more general, maybe I should have added that. I meant overall any site with valuable info. Re prediction markets yes, regulation is 100% required. So many people just lose...
Oh my comment is a lot more general, maybe I should have added that. I meant overall any site with valuable info. Re prediction markets yes, regulation is 100% required. So many people just lose their fortunes in that stuff and there's a ton of cheating involved, idk why people do it but its a popular thing.
The Internet has always been a hostile place but it's becoming more so. I think we're going to see many websites either going behind a proxy like Cloudflare, or converting into a static website so that the web pages can be served by a CDN.
The numbers here were really eye-opening read for me. Yea I've heard the headlines about dead internet theory, bots > than humans, and the prevalence of crawling with ai, but I really had no idea of the scale, so seeing the ratio of 38,000:1 anthropic crawls:visitor was honestly shocking.
Also really interesting to see how, in the case of The Numbers, they're treating the crawlers as the audience to increase paid services.
Flip side, though: this really highlights how irresponsible the ai companies comes across to me, releasing such powerful tools into the wild and only then playing whackamole to put up guardrails and fix problems that could reasonably have been anticipated (though I'll grant, devious people can be clever as hell).
I’ve been saying that ChatGPTs Release without many many more disclaimers was incredibly reckless. The amount of people not realizing that just because it’s written well, it doesn’t mean that it’s correct. Is astounding.
Especially early models. Like I’ve had a friend who’s ChatGPT confirmed his conspiracy theories like the moon landing being fake and others
Ah, of course the underlying reason was prediction markets. I figured there had to be some financial reason why anyone would bother throwing so much traffic at the site. Simply scraping answers is probably a good share too, but that wouldn't explain the level of dedication to rapid data access.
The only saving grace is private and niche communities that is hard to craw or not designed for SEO. Back in the day, every website was designed to be indexed by Google but with today's tech we can actually do the opposite and make it as hard as possible for AI bots to crawl your site. Cloudflare is already working on this.
Any website with great content will eventually wall up. It just makes sense cuz otherwise you'd be giving it free to OpenAI and Anthropic and basically let them make money while you suffer.
Funny how the internet started to be as open as possible but actually now the best way to build a community is to make it as close and as niche as possible.
Without proper regulation around this stuff, there no other way to operate sanely. Unless you are fine spending yours (or someone else's) full-time efforts fighting bots.
Prediction markets are 100% gooig to come under fire in the coming few years, so I guess in this specific instance that will bring relief here.
Oh my comment is a lot more general, maybe I should have added that. I meant overall any site with valuable info. Re prediction markets yes, regulation is 100% required. So many people just lose their fortunes in that stuff and there's a ton of cheating involved, idk why people do it but its a popular thing.