Earlier this month, one of the websites we look after started to slow down. Pages were loading slowly, and other websites on the same server were starting to feel it too. There was no sale on and no rush of customers. When we opened the server logs, the cause was obvious: crawlers, mostly Meta's AI crawlers, were working through thousands of variations of the same few pages.
We had it under control the same day. But it made us ask a bigger question. If AI crawlers can do that to one site, how much of the crawling across all the websites we host is now coming from AI?
So we counted. We took a sample of live business websites we host and analysed every request from an automated crawler over 14 days, 1.4 million crawler requests in total. Then we sorted each one by who sent it.
"We've spent twenty years treating Google as the one visitor that matters most. Our own logs now show AI systems reading our clients' websites far more often than Google does. That changes what a website is for." Giedrius Kudzinskas, founder of Rubik Digital
How much crawler traffic comes from AI?
AI crawlers made 582,806 of the 1,440,971 crawler requests we analysed. That's 40.4%, or two in every five. Traditional search engines, including Google, Bing, Apple and Baidu, made 28.5% between them. AI crawlers made 42% more requests than all of them put together.
| Type of crawler | Requests | Share of crawler traffic |
|---|---|---|
| Search engines | 410,870 | 28.5% |
| AI search and assistant crawlers | 389,137 | 27.0% |
| Other bots, such as uptime monitors and link previews | 287,769 | 20.0% |
| AI training crawlers | 193,669 | 13.4% |
| SEO tools, such as Ahrefs and Semrush | 159,526 | 11.1% |
Put the two AI rows together and AI is the largest group of crawlers by a clear margin.
Do AI crawlers outnumber Googlebot?
Yes, and by a long way. Googlebot made 20,288 requests in the same fortnight. AI crawlers made 28.7 times as many.
That headline figure is pushed up by our three busiest sites, so we checked it without them. The AI crawlers still made 8.7 times as many requests as Googlebot. On almost nine in ten sites, AI crawlers made more requests than Googlebot. The only exceptions were two of the quietest sites in the sample.
The mix varied a lot from site to site. AI made 0.2% of crawler traffic on one site and two-thirds on another. On the typical site, it was about 12%.
Volume isn't the same as value. Google still sends far more customers to these sites than any AI tool does. But the crawl numbers show where the attention is going. AI companies are building their own indexes of the web, fast, and they come back again and again to keep them fresh.
Which AI crawlers visit websites the most?
Two companies dominate. Meta made 57% of AI crawler requests and OpenAI made 38%. Everyone else, including Anthropic, Amazon and Perplexity, shared the last 5%.
| Crawler | Company | What it's for | Requests |
|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search results | 203,539 |
| meta-externalagent | Meta | Training Meta's AI models | 167,207 |
| meta-webindexer | Meta | Meta AI search results | 166,845 |
| Amazonbot | Amazon | Amazon's AI services and Alexa | 13,521 |
| ChatGPT-User | OpenAI | Reading a page live when someone asks ChatGPT about it | 11,318 |
| GPTBot | OpenAI | Training OpenAI's models | 5,935 |
| ClaudeBot | Anthropic | Training Claude | 4,773 |
| PerplexityBot | Perplexity | Perplexity search results | 2,157 |
| Claude-User | Anthropic | Reading a page live when someone asks Claude about it | 1,789 |
| Claude-SearchBot | Anthropic | Claude's web search | 1,318 |
One detail stood out. Two-thirds of AI crawling was for search and answers. Only a third was for training. The common picture of AI companies scraping the web to build models is now only part of the story. Most of the crawling we saw was AI tools getting ready to answer questions about these businesses.
Some of it was happening in real time. ChatGPT and Claude fetched pages on behalf of a user more than 13,000 times in two weeks, about 940 times a day. These requests show AI assistants retrieving website content in connection with user interactions. Unlike traditional crawling, some of this activity happens when an AI assistant needs information to answer a question. It creates an opportunity for businesses to be discovered without the user necessarily visiting their website.
"Some of those live fetches will be potential customers doing their research through ChatGPT instead of Google. They might never land on your website at all. The AI reads it for them and tells them what it found." Giedrius Kudzinskas, founder of Rubik Digital
Can AI crawlers slow down your website?
They can, and we've seen it happen. 1.6% of AI crawler requests in our data ended in a server error, which usually means the site was too busy to respond.
On the site that started this study, the problem was filter pages. An online catalogue can produce thousands of web addresses from the same few pages, one for every combination of filters. People never click through all of them. Crawlers do. Every one of those requests made the server do real work, and real visitors had to wait behind them.
We took a targeted approach rather than shutting AI out. We blocked Meta's training crawler, which was doing the most damage and brings the site no visitors. We stopped Meta's search crawler from requesting the filter pages, and told all crawlers to skip those pages in the site's robots.txt file. Then we capped how much server capacity that one site could use, so it could never slow down anyone else again. Google, ChatGPT, Claude and the other AI search tools can still reach every page that matters.
Should you block AI crawlers?
For most businesses, no. Blocking AI search crawlers is a quick way to disappear from the answers that ChatGPT, Claude and Perplexity give. Training crawlers are a separate decision, and blocking them won't affect your Google rankings at all.
| Crawler type | Examples | Our advice |
|---|---|---|
| AI search and live fetches | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot | Allow them. They decide whether AI tools can recommend you. |
| AI training | GPTBot, ClaudeBot, meta-externalagent, CCBot | Your call. Blocking them doesn't affect Google, but your content won't shape future models. |
| Any crawler overloading your server | Any of the above | Slow it down rather than shut it out. |
"I'd rather a client's website is easy for AI to read than locked away from it. You can't be recommended by something that can't see you. If a crawler is causing trouble, slow it down. Don't shut the door." Giedrius Kudzinskas, founder of Rubik Digital
How can you see AI crawler traffic on your own website?
Google Analytics won't show it. Crawlers don't run the tracking code, so they never appear in your reports. You need to look in one of these places instead:
- Your server logs. Ask your developer or hosting company to search them for the crawler names in the table above.
- Your CDN dashboard. If your site uses Cloudflare, the AI Crawl Control section lists which AI crawlers visit and lets you allow or block each one.
- Your robots.txt file. Check you aren't blocking AI search crawlers by accident. Broad rules copied from old templates are a common cause.
Google Analytics is still useful for the other half of the picture: the people AI tools send to you. Look for the AI Assistant channel, or referrals from chatgpt.com, perplexity.ai and claude.ai.
If AI crawlers are visiting but you never come up in AI answers, the issue is usually what's on the page rather than access. Our guide to appearing in Google AI Overviews and AI Mode covers what to change.
How we did the study
We analysed server logs from a sample of 17 live business websites that we host and manage, from 28 September to 11 October 2026. The sites include finance, travel, healthcare, construction, leisure, sport and professional services.
We only looked at requests from automated crawlers that identify themselves by name, such as Googlebot or GPTBot. Requests from people visiting the sites were filtered out and not analysed. We counted requests for pages only, not for images, scripts, fonts or other files. Requests that were blocked or slowed down before they reached the website aren't included, and that could affect the share of any crawler category, not just AI.
Each crawler was classified by the name it declares in its user-agent string. We didn't independently verify every crawler's identity, for example with reverse DNS lookups, so a small share of requests may come from bots using another crawler's name.
We've only published combined figures. No client is named and no information about website visitors was used.
Want to know how AI tools see your website? Our SEO team can check your crawler access, your robots.txt and how you show up in AI answers. Get in touch.
