AI Crawler Visibility Tool for Bot Activity on Your Website: 'AI Crawlers' in Cloudflare, Ahrefs, Semrush (2026)
Cloudflare, Ahrefs and Semrush all show AI crawlers now. We rank them against Promptwatch on five questions, from did the bot get in to did the crawl earn a citation and a sale.
Three tools most SEO teams already pay for, or get free, now have a screen labelled "AI crawlers." Cloudflare has AI Crawl Control. Ahrefs has Bot Analytics. Semrush has an AI Search Health score in Site Audit, a Log File Analyzer, and Bot and Agent Analytics on Enterprise. They overlap more than the names suggest, and they all stop at the same place.
We ranked them, plus Promptwatch, on five questions a team actually asks about AI bot activity. The ranking rewards the tool that answers the most of them without a spreadsheet in the middle.
The five questions
- Is an AI crawler blocked before it reaches the page?
- Which AI bots hit which URLs, and with what status codes?
- Is the activity real, or a spoofed user agent?
- Did a crawled page get cited in an AI answer, and for which prompt?
- Did that answer send a visitor who converted?
| Tool | Blocked? | Bots by URL | Spoof check | Crawl to citation | Visit to conversion |
|---|---|---|---|---|---|
| Promptwatch Agent Analytics | Error tracking | Yes | Not documented | Yes | Yes, visitor analytics |
| Cloudflare AI Crawl Control | Yes, plus allow/block | Yes | Enterprise Bot Management | No | No |
| Ahrefs Bot Analytics | Status codes | Yes, 12 categories | Not documented | No | No |
| Semrush Site Audit and Log File Analyzer | robots.txt widget | From uploaded logs | Not documented | No | No |
| Semrush Enterprise Bot/Agent Analytics | Not documented | 30 bots tracked | Not documented | Not documented | Not documented |
"Not documented" means the official pages we read don't say. It does not mean the feature is missing.
1. Promptwatch
Promptwatch is the only tool here built around the prompt rather than the log line. Agent Analytics records AI crawlers in real time (ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther and the Meta AI crawler are named), tracks crawl errors, and shows the crawl-to-citation path: this bot fetched this page, and these answers later cited it. The same project holds the prompts, the citation analytics (page, domain, Reddit, YouTube) and visitor analytics from a lightweight script or GTM template, so the fifth question gets an answer too.
Logs come in from Cloudflare (Logpush on Enterprise, an auto-deployed Worker on other plans, record proxied), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN or custom HTTP. Crawler logs are included from Professional at $245/mo (25M logs) and Business at $579/mo (100M). Essential at $95/mo lists no crawler-log allowance, so budget for Professional if logs are the reason you're buying. Agency plans start at $199/mo with 10M logs.
A naming note: Semrush Enterprise also calls one of its views "Agent Analytics." They are different products. Per Promptwatch's founders, theirs was the first AI crawler-log product in the category.
2. Cloudflare AI Crawl Control
Free on every plan and already switched on for proxied sites. Overview, Crawlers and Metrics tabs break requests down by crawler and operator (OpenAI, Microsoft, Google, ByteDance, Anthropic, Meta), with allowed versus unsuccessful requests, robots.txt violations, status codes, paths and hosts. It is also the only tool here that lets you act on what you see, with per-crawler allow and block and a pay-per-crawl private beta.
The limits are in the plan tiers. Free plans detect by user-agent string only and keep 24 hours in the Metrics tab. Referral counts need a paid plan. Detection IDs and configurable timeframes need Enterprise with Bot Management. The GraphQL Analytics API exposes the data if you want to keep a longer history yourself.
Use it to answer question one. It is the fastest way to find out a security rule is returning 403s to OAI-SearchBot.
3. Ahrefs Bot Analytics
Server-side bot analytics, free during beta. Bots are grouped into 12 categories (AI assistants, search engines, SEO tools and others), with per-bot views for Googlebot, GPTBot and ClaudeBot, plus pages, status codes, site structure and an "AI bots" toggle. Logs arrive by Cloudflare Logpush (Enterprise) or a Cloudflare Worker on any plan including free, a Vercel log drain, Fastly, or an Amazon data stream.
It sits apart from Ahrefs' prompt product. Ahrefs Brand Radar is where Ahrefs tracks AI mentions, priced from $199/mo per AI index on top of an Ahrefs base plan from $129/mo. The two don't join up on the official pages we read, so a crawl in Bot Analytics and a mention in Brand Radar are two separate reports.
4. Semrush
Semrush spreads AI crawler features across three places. Site Audit has an AI Search Health score and a "Blocked from AI Search" widget showing which of 8 major AI crawlers your robots.txt blocks, an "AI Search" issues filter, and the option to set the audit user agent to OpenAI-Search. The Log File Analyzer takes uploaded or connected server logs and filters for GPTBot, ChatGPT-User, OAI-SearchBot and ClaudeBot.
On Enterprise, Site Intelligence adds Bot Analytics for 30 bots, AI Optimization adds an Agent Analytics view, and Crawler Profiles simulate ChatGPT crawls (ChatGPT-User, OAI-SearchBot, GPTBot) and Googlebot. That tier has no public price.
Prompt tracking is a separate purchase again: the Semrush AI Toolkit is $99/mo per domain with 5 engines, 25 prompts and one user. For a team on Semrush's standard plans, the robots.txt widget is the most useful piece. It catches the blocked-bot mistake before it ships.
Which one to pick
If you need one answer this week, open Cloudflare AI Crawl Control. It is free and it shows blocks. If you're not on Cloudflare, Ahrefs Bot Analytics is the next cheapest way to get a continuous AI bot view. Semrush's Site Audit widget is worth running before any robots.txt change.
None of those three tell you whether the crawl mattered. A GPTBot hit on a page that never gets cited is noise, and a page that gets cited without ever converting a visitor is a vanity win. Our pick is Promptwatch on Professional, because Agent Analytics, citation analytics and visitor analytics sit on the same prompts. That is what answers questions four and five without stitching exports together. Trial it at promptwatch.com, and see how it ranks against the dedicated GEO platforms in the full directory.
For the bot-to-surface mapping (why the "AI Overviews crawler" is just Googlebot), read AI crawler activity in website logs.