Bots account for almost a third of web traffic

Bots have crawled the web for a long time, but the past couple years has been something different as companies release their AI crawlers to scrape as much as possible. Cloudflare broke it down by type of bot and source.

Not all crawlers are the same. Bots, automated scripts that perform tasks across the Internet, come in many forms: those considered non-threatening or “good” (such as API clients, search indexing bots like Googlebot, or health checkers) and those considered malicious or “bad” (like those used for credential stuffing, spam, or scraping content without permission). In fact, around 30% of global web traffic today, according to Cloudflare Radar data, comes from bots, and even exceeds human Internet traffic in some locations.

A new category, AI crawlers, has emerged in recent years. These bots collect data from across the web to train AI models, improving tools and experiences, but also raising issues around content rights, unauthorized use, and infrastructure overload. We aimed to confirm the growth of both search and AI crawlers, examine specific AI crawlers, and understand broader crawler usage.

Every now and then I glance at traffic sources, and AI bots seem increasingly common. I wonder if or when bot traffic outnumbers human visits.