In August 2026, 35.6% of traffic on Cloudflare's global network came from machines talking to machines. Not humans browsing. Not apps refreshing feeds. Bots. The number understates reality because Cloudflare blocks roughly 6% of global traffic as malicious and millions of its customers have turned on AI crawler blocking. The raw share is higher.
Three cloud providers originate nearly a third of all bot traffic. Amazon at 14.7% across two AS numbers. Microsoft at 9%. Google at 8.5%. If you want to find the AI that reads the internet, follow the cloud provider AS numbers. The United States originates 42.6% of all bot requests. The infrastructure is concentrated. The consequences are distributed.
These numbers are from Cloudflare Radar, measured across a network of 330 cities handling 81 million HTTP requests per second. They are the most current public measurement of bot traffic on the internet.
Most of this traffic is not AI. The Cloudflare 2025 Year in Review found that search engine crawlers made up 40% of verified bot traffic. AI crawlers accounted for 20%. SEO bots, the commercial scraping infrastructure that monitors rankings and backlinks, made up 13%. Googlebot alone generated more than a quarter of all verified bot traffic and 4.5% of HTML requests. All other AI bots combined generated 4.2%.
The fastest-growing category is not any of these. It is user-action crawling. A human asks a chatbot a question. The chatbot visits a web page to answer it. The human reads the answer without seeing the page. That category grew 15x in 2025.
The widely cited claim that bots account for more than half of web traffic originated with Imperva's 2016 Bad Bot Report, which examined 16.7 billion visits to 100,000 domains and found 52% automated traffic. Imperva's 2023 update put the figure at 49.6%, partly attributing the rise to AI models scraping the web. These numbers became the statistical backbone of the dead internet theory. For years the theory was dismissed as conspiracy. Then the data caught up.
When a conspiracy becomes a measurement
The dead internet theory originated in a January 2021 post by a user named IlluminatiPirate on the forum Agora Road's Macintosh Cafe. The post argued that the internet had become a wasteland of automated content, manipulated by algorithmic curation at the hands of coordinated state actors. The conspiratorial element made the theory easy to dismiss. The observable element kept getting harder to ignore.
Kaitlyn Tiffany brought the theory into mainstream coverage with a September 2021 article in The Atlantic. In 2024, Yoshija Walter argued in AI and Society that the "once speculative theory" was becoming observable as AI-generated content proliferated. In 2026, Hal Berghel published in Computer what he called a "leaner" version of the theory, "stripped of paranoia, prejudice, politics and polemic." His central argument: dismissing the core observations because conspiracy theorists attach implausible claims to them is "a mistake of the first order."
Then the theory escaped academic journals and entered CEO Twitter feeds. In May 2025, Bloomberg reported that tools like ChatGPT and Midjourney had made the theory "look downright prophetic." In September 2025, Sam Altman posted on X that "there are really a lot of LLM-run twitter accounts now." That same month, Time magazine covered the theory's shift from fringe to mainstream.
In October 2025, Forbes reported that Ohanian and Altman had both warned the dead internet theory was materializing. Reddit co-founder Alexis Ohanian told Kevin Rose at TechCrunch Disrupt that "the dead internet theory is real." The trajectory from fringe post to CEO admission took 4 years.
Along the way, the platforms started building the thing the theory described.
In December 2024, Meta announced AI-powered autonomous accounts on Facebook and Instagram. Connor Hayes, VP of product for generative AI, told the Financial Times these AI characters "will have bios and profile pictures and be able to generate and share content powered by AI on the platform." The accounts were removed after backlash. The headline had already written itself: the company was building bots to populate its own social network.
The same year, Facebook was already flooded with what came to be called AI slop. AI-generated images of flight attendants, children next to artwork they supposedly painted, and thousands of variations of "Shrimp Jesus", Christ merged with shellfish, collected hundreds of thousands of comments saying "Amen." NBC News reported the comments flooding in by the thousands. Bots generated the content. Humans engaged with it. The feedback loop the theory had predicted was running.
In January 2026, Ohanian and Kevin Rose relaunched Digg. It shut down 2 months later, citing an "unprecedented bot problem." In May 2026, Digg rebranded as an AI news aggregator, stating on its homepage that bots were the reason for the pivot.
How crawlers work, and how they lie
Web crawlers have existed since 1994, when Martijn Koster proposed the Robots Exclusion Protocol after his server was overwhelmed by a poorly written crawler. The protocol has been voluntary for 32 years. A website puts a robots.txt file on its server. A crawler is supposed to read it and obey. Nobody enforces it. RFC 9309, published in 2022, states explicitly that robots.txt rules "are not a form of access authorization." It is a request.
Every major AI company publishes documentation for its crawlers. OpenAI lists 4: GPTBot for training, OAI-SearchBot for search, ChatGPT-User for user-driven visits, and OAI-AdsBot for ad verification. Google publishes Googlebot for search and AI training combined, plus Google-Extended as an opt-out for Gemini training. Perplexity lists PerplexityBot for search and Perplexity-User for user visits. Anthropic's documentation sits behind a support page.
The documentation is public. Compliance is optional.
In July 2024, Cloudflare discovered bot operators spoofing browser user agents to evade detection. The company's machine learning models caught the pattern anyway. A specific bot, identified by researchers as Perplexity AI, was observed hiding behind Chrome user agent strings while rotating IPs across different AS numbers. Perplexity's declared crawler made 20 to 25 million daily requests. Its stealth crawler added another 3 to 6 million. Cloudflare de-listed PerplexityBot from its verified bot directory.
OpenAI's ChatGPT-User, by contrast, fetched robots.txt and complied with crawl restrictions. Voluntary compliance is technically possible when operating in good faith.
The deeper structural problem is Googlebot. Cloudflare's September 2025 AI bot principles blog identified the issue: Googlebot crawls for both search indexing and AI training in a single pass. A website that wants to appear in Google Search must allow Googlebot. That same crawl feeds Google's AI models. Google-Extended, the opt-out for Gemini training, explicitly does not apply to AI Overviews, which are attached to search. As Cloudflare put it, the combined crawler "forces an impossible choice onto website owners."
OpenAI separates its crawlers by purpose. A publisher can allow OAI-SearchBot and block GPTBot. Google offers no equivalent separation.
The 70,900 to 1 problem
In July 2024, Cloudflare published its first network-wide survey of AI crawler traffic. The top 4 crawlers by request volume were Bytespider from ByteDance, Amazonbot, ClaudeBot from Anthropic, and GPTBot from OpenAI. Bytespider accessed 40.40% of Cloudflare-protected websites. GPTBot accessed 35.46%. ClaudeBot reached 11.17%. Among the top 1 million internet properties, 38.73% were accessed by AI bots and only 2.98% took action to block them. Among the top 10 sites by traffic, 80% were accessed and 40% blocked.
The 2025 data shifted. Googlebot's crawl volume dwarfed every other AI bot. Anthropic's ClaudeBot crawling doubled in the first half of 2025 before declining. PerplexityBot grew 3.5x. Bytespider continued the decline it started in 2024.
The most revealing metric Cloudflare introduced in 2025 was not about volume. It was about value. In July 2025, Cloudflare launched the crawl-to-refer ratio: for every page an AI platform crawls, how many visitors does it send back.
The ratio broke the basic transaction that funded the web for 3 decades. Search engines crawl content. They send traffic. That was the deal. Google crawled at roughly 3 pages per referred visitor in early 2025. DuckDuckGo stayed below 1:1. Bing ranged between 50 and 70:1.
AI platforms operated in a different universe. Anthropic's ratio was 70,900 to 1. For every 70,900 HTML pages Anthropic's crawlers requested, Claude sent 1 visitor to a source site. OpenAI peaked at 3,700 to 1 in March 2025, falling as ChatGPT search gained usage. Perplexity, the most referral-friendly AI platform, stayed below 200 to 1 from September 2025 onward.
Cloudflare's CEO wrote that getting traffic from OpenAI is 750 times harder than it was from Google in the previous decade. Getting traffic from Anthropic is 30,000 times harder. On 1 July 2025, Cloudflare declared "Content Independence Day" and announced it would default to blocking AI crawlers that do not pay creators.
What publishers see when the traffic stops
Pew Research Center published data in July 2025 from 68,879 Google searches across 900 US adults. Users who saw an AI-generated summary clicked on a traditional search result 8% of the time. Users who did not see a summary clicked 15% of the time. Users clicked on a link within the AI summary itself in 1% of visits.
The same study found that 18% of Google searches produced an AI summary in March 2025. Wikipedia, YouTube, and Reddit supplied 15% of the sources inside those summaries. Government websites appeared more often in AI summaries than in standard results.
This did not start with AI. Google's zero-click search rate on mobile had already hit 75% before AI Overviews launched. 3 out of 4 mobile searches were answered without leaving Google. AI Overviews, launched in May 2024, extended the pattern. Google was already sending less traffic. AI summarization sent even less.
In March 2024, Google acknowledged its search results were being flooded by websites that "feel like they were created for search engines instead of people." The company's blog post on its March 2024 core update promised to reduce "low-quality, unoriginal content" in search results by up to 40%. The AI that was scraping the web was also filling it with content designed to be scraped.
The protocols try to catch up
The IETF formalized robots.txt as RFC 9309 in September 2022. The standard explicitly states it is "not a form of access authorization." It is a request. In April 2026, the IETF AI Preferences Working Group published draft-ietf-aipref-vocab-06, defining 2 categories: Train-AI for model training and Search for indexed content. The draft is not yet a standard. It is the first serious attempt at a machine-readable preference layer more granular than robots.txt.
Cloudflare shipped faster than the IETF. It launched 1-click AI crawler blocking in July 2024, AI Crawl Control analytics in September 2024, and a Pay Per Crawl marketplace in July 2025. The marketplace lets publishers set a price per 1,000 tokens for AI crawler access, authenticated through Cloudflare's verified bot framework.
The Internet Archive has crawled and preserved the web since 1996. Its crawler obeys robots.txt and does not train AI models. But its crawler consumes the same server resources as AI training crawlers. A well-behaved bot and a training bot impose the same infrastructure cost. The difference is downstream.
The browser that is also an agent
The next phase will make text crawlers look efficient. Anthropic announced Claude computer use in October 2024, enabling Claude to view screens, move cursors, click buttons, and type. Google launched Project Mariner in December 2024, a Chrome extension that controls the browser to complete tasks. The open-source Browser Use framework lets any AI agent navigate websites programmatically.
A text crawl fetches 2 or 3 HTTP requests. A browser agent loads every image, every JavaScript bundle, every tracking pixel, every analytics call. It renders the page, screenshots it, and feeds the image to a vision model. A single agent browsing session generates hundreds of requests across CDNs, analytics services, and ad networks. The resource consumption of 1 agent browser visit may equal 100 traditional crawler requests.
The economic relationship is even more one-sided. A traditional crawler indexes content for retrieval. A browser agent consumes content to perform a task: book a flight, file a ticket, check a price. The website serves as infrastructure for the agent. No human visits. No brand impression. No subscription. No ad revenue. The site operator pays the compute bill for a transaction that generates revenue for someone else.
The agents are already browsing.
Who pays for the web when machines do the reading
A website that serves a billion pages to humans generates revenue from the humans who arrive. A website that serves a billion pages to AI crawlers generates infrastructure costs.
The Cloudflare Pay Per Crawl marketplace is the first attempt at a settlement layer. It charges per content type: news, research, reference. It uses verified bot identity to authenticate who is crawling. Whether it works depends on whether AI companies find paying cheaper than the legal risk of crawling without payment.
The legal risk is real. The New York Times sued OpenAI and Microsoft in December 2023 for copyright infringement through training data ingestion. The Chicago Tribune, Denver Post, and multiple Alden Global Capital publications have filed similar suits. The outcome will determine whether copyright law extends to the ingestion layer or treats AI training as a category outside existing frameworks.
Google's AI Overviews, launched in May 2024, generate summaries at the top of search results. Google claims the summaries include links to sources and send more traffic to publishers than traditional listings. Publishers dispute this, pointing to aggregate declines in organic search traffic since the feature launched. Both sides have data. Neither has a settlement mechanism.
The open data paradox
Common Crawl is a non-profit that maintains a free, open repository of web crawl data used extensively for AI training. Its crawler, CCBot, obeys robots.txt and supports crawl-delay. The foundation published dedicated IP ranges for verification in November 2024. Its July 2026 archive contains 2.14 billion web pages, 364 TiB of uncompressed content. Its web graph for May to July 2026 spans 240.4 million nodes and 3.7 billion edges. The foundation also published an AI Visibility Audit for publishers.
The paradox Common Crawl embodies is the paradox of the whole situation. A non-profit runs a crawler that obeys robots.txt and publishes data openly. Commercial AI companies use that same data to train models that crawl 70,900 pages for every 1 they refer. The crawler is the same mechanism. The downstream effects are not.
Bots have always been here
The web has had robots since 1994. The robotstxt.org resource defines them as "programs that traverse the Web automatically." In 2017, The Atlantic published "The Internet Is Mostly Bots." In 2018, New York Magazine ran "How Much of the Internet Is Fake? Turns Out, a Lot of It, Actually," cataloguing fake YouTube views, fake social media followers, and programmatic ad fraud.
The difference between 2018 and 2026 is what the bots are doing. The 2018 bots sold fake engagement. They inflated view counts, generated fake likes, and stole ad dollars through fraudulent impressions. They were parasites on the attention economy. The 2026 bots consume real content at industrial scale to train models that make the attention economy unnecessary. A fake YouTube view did not reduce the value of a real one. An AI summary that replaces the click does.
The dead internet theory, for years treated as a fringe conspiracy, has proven to be the framework that most closely maps onto the observable shift. In its lean form, stripped of the conspiracy, the diagnosis is straightforward: automated content is displacing human-authored content, and the economic and social structures built for human-to-human connection were not designed for this transition.
The issue is not that bots outnumber people. The issue is that the protocols, norms, and business models of the open web never anticipated a world where machines consume content at industrial scale and refer almost nothing back.
All data was verified in August 2026. The Cloudflare Radar bot traffic share of 35.6% was observed on 12 August 2026. The 2025 Year in Review covers through 2 December 2025. IETF drafts are current as noted. All sources are linked inline.