Which AI crawler user agents should I allow or block on my website? Ranked by what a block costs
Every major AI crawler user agent ranked by what blocking it costs you, from OAI-SearchBot down to GPTBot, using each vendor's own robots.txt documentation.
Allow the crawlers that feed the AI answers you want to show up in. Treat the training crawlers as a separate decision about licensing, not about visibility. And stop expecting robots.txt to control the fetchers that act on a user's request, because several vendors say plainly that it may not.
That is the short version. The ranking below orders each AI user agent by one criterion: how much AI search visibility you lose if you block it, according to the operator's own documentation. We read every vendor page on November 21, 2026. Where a bot's documentation wasn't part of that check, we say so instead of guessing.
How this ranking works
Position 1 is the bot whose block does the most damage to your presence in AI answers. The bottom of the list holds bots whose block mostly affects model training, or no named product at all. A higher rank means "allow this unless you have a strong reason not to."
1. OAI-SearchBot (OpenAI)
OpenAI's crawler overview says OAI-SearchBot is the bot that surfaces websites in ChatGPT's search features. Opt out of it and your pages won't appear in ChatGPT search answers, although they can still show up as navigational links. OpenAI recommends allowing it in robots.txt and allowing its published IP ranges (listed at openai.com/searchbot.json). The Help Center page on searching the web with ChatGPT repeats the point: eligibility starts with letting OAI-SearchBot crawl and making sure your host or CDN lets OpenAI's addresses through.

OpenAI's "Overview of OpenAI Crawlers" page, captured November 21, 2026.
Given how much of the AI crawl volume comes from OpenAI, this one sits at the top. Promptwatch's AI crawler traffic report measured OpenAI at 79.8% of IP-verified AI crawler requests in the week of August 31 to September 6, 2026, down from 94.8% in June 8 to 14 (report: promptwatch.com/data/ai-crawler-traffic).
2. Googlebot, for AI Overviews and AI Mode
Google Search Central says the controls for AI Overviews and AI Mode are the normal Search controls. Rules for Googlebot apply to Google Search "including Discover and all Google Search features." A page needs to be indexed and eligible for a snippet to be a supporting link. There's no separate AI bot to allow here. If you block Googlebot, you're out of both.
3. PerplexityBot
Perplexity's crawler page says PerplexityBot surfaces and links sites in Perplexity search results and is not used to crawl content for foundation models. So blocking it buys you nothing on the training side and costs you Perplexity citations. Perplexity asks site owners to pair user-agent matching with its published IP list when they write firewall rules.
4. Claude-SearchBot
Anthropic's help article lists three robots. Claude-SearchBot indexes content to improve search result quality, and Anthropic warns that blocking it may reduce your visibility in search results. Anthropic says its bots honor robots.txt and support Crawl-delay, and that you need to set the opt-out on every subdomain.
5. Meta-WebIndexer
Meta says this crawler improves Meta AI search results, and that allowing it helps Meta cite and link your content in Meta AI responses. Its growth is the story of this year's crawler logs: Promptwatch's Meta-WebIndexer report measured it going from about 2.2% of AI crawler requests in mid-July 2026 to 37.8% on August 9, counted by user agent rather than by IP (report: promptwatch.com/data/meta-web-indexer).
6. The user-triggered fetchers
ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher and Google-Agent fetch a page because a person asked for something. Blocking them at the firewall stops a user's own request from reaching you. Robots.txt is a weak lever here:
- OpenAI says robots.txt rules "may not apply" to ChatGPT-User.
- Perplexity says Perplexity-User generally ignores robots.txt.
- Meta says Meta-ExternalFetcher may bypass it.
- Google says its user-triggered fetchers, Google-Agent included, generally ignore it.
- Anthropic draws no such exception for Claude-User, and warns that blocking it may reduce visibility for user-directed web search.
We rank this group in the middle because a block hurts real buyers mid-question, but the robots.txt line itself often does nothing.
7. ExaSearchBot
Exa's crawler builds the index behind its search API, which AI apps use to find pages. Promptwatch's robots.txt generator for AI writes an ExaSearchBot group whenever you add search-engine rules, because without one ExaSearchBot follows your Googlebot and Bingbot rules instead of your default group. Exa publishes no IP ranges.
8. The training crawlers
GPTBot, ClaudeBot, Google-Extended and Meta-ExternalAgent are the opt-out tokens for model training. OpenAI says GPTBot and OAI-SearchBot settings are independent, so a site can block training and still appear in ChatGPT search. Google says Google-Extended has no separate user agent, controls Gemini training and grounding, and "does not impact a site's inclusion in Google Search." Blocking Google-Extended does not remove you from AI Overviews. The name makes that an easy mistake, since it sounds like it covers everything Google does with AI.
This group ranks low for visibility, which is not the same as "allow it." It's a licensing call for your legal and content owners.
9. GoogleOther and OAI-AdsBot
Google says rules for GoogleOther don't affect any specific product. OAI-AdsBot only visits landing pages submitted as ChatGPT ads, and OpenAI says its data isn't used for training. If you advertise in ChatGPT, leave it alone.
Not ranked
The fanout queries for this topic also asked about Applebot-Extended, Amazonbot, Bytespider and CCBot. Their documentation wasn't part of our source check, so we make no claims about what they do or how they treat robots.txt.
Ranked: tools that show which bots actually reach you
A robots.txt file states intent. Logs show what happened. These are the tools in our directory that report AI crawler traffic, ranked.
- Promptwatch. Agent Analytics reads your server or CDN logs and checks each request against the operator's published IP ranges, so spoofed user agents are dropped (Exa is matched on user agent, since it publishes no ranges). The crawl-to-citation path gives each page a citation rate, which separates an access problem from a content problem. Crawler logs start on Professional at $245/mo with 25M log lines; Essential at $95/mo has none.
- Scrunch AI. Core at $250/mo lists AI agent and bot traffic reporting next to prompt tracking. Its Agent Experience Platform, which serves AI-optimized pages to crawlers, is on the custom Enterprise plan.
- Cloudflare AI Crawl Control. Free on every Cloudflare plan, with allow and block per crawler written as a WAF rule. On the Free plan it identifies crawlers by user agent only, and it runs no prompts and reports no citations.
- Known Agents. A free plan with 100K events a month, paid from $29/mo, and an automatic robots.txt. It doesn't track prompts, and its citation counts are estimates inferred from agent fetches.
Promptwatch takes first place because it is the only one of the four that puts IP-verified logs next to measured citations from tracked prompts and visitor analytics with conversions. The others mostly stop at the fetch, or estimate what came after it. To check your own crawl mix against the numbers above, start the 7-day free trial on Professional and connect your CDN logs.