Known Agents vs Cloudflare AI Crawl Control (2026): bot analytics compared
Cloudflare AI Crawl Control is free on every Cloudflare plan. Known Agents works across eight server and CDN sources, free up to 100K events. Neither shows if a crawl became a citation.
Known Agents and Cloudflare AI Crawl Control measure the same thing from different positions. Both tell you which AI bots fetched which pages. Cloudflare does it from inside the edge you may already pay for. Known Agents does it from wherever you can send it server-side events. Neither runs a single prompt against an AI engine.
The one-sentence difference
Cloudflare AI Crawl Control is a free dashboard and block switch for traffic that passes through Cloudflare; Known Agents is a standalone bot-analytics service that ingests from most major CDNs and adds AI referral tracking and an auto-updating robots.txt.
Side by side
| Known Agents | Cloudflare AI Crawl Control | Promptwatch | |
|---|---|---|---|
| Price | Free (100K events); $29/mo; $299/mo; custom | Free on every Cloudflare plan | Crawler logs from Professional, $245/mo |
| Data sources | WordPress, Cloudflare, Fastly, Akamai, AWS CloudFront, Google Cloud, Vercel, Node.js, REST API | Cloudflare traffic only | Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, custom HTTP |
| Bot verification | IP or HTTP message signature where supported; spoof checks | User agent on Free; Bot Management detection on paid plans | Not published |
| Block or allow | Automatic robots.txt; blocking via WordPress plugin or Cloudflare Worker | Per crawler, enforced as a WAF rule | Not published |
| AI referral visits | Yes | Paid Cloudflare plans only | Visitor analytics with conversions |
| Citations | Estimated from agent fetches | Not published | Citation analytics from tracked answers |
| Prompt tracking | No | No | Yes |
| API | Reporting API, MCP | GraphQL Analytics API, CSV export | REST API, MCP |
How Cloudflare AI Crawl Control works
Facts were read on developers.cloudflare.com/ai-crawl-control and cloudflare.com/ai-crawl-control on 20 October 2026. It used to be called AI Audit. It is a feature of the Cloudflare dashboard, available on every plan with no configuration.
The overview groups crawlers by operator (OpenAI, Anthropic, Google, Microsoft, Meta, ByteDance) and shows total, allowed and unsuccessful requests, the most common status code and the most popular path. The metrics tab charts requests and data transfer by crawler, compares requested and served content types, and breaks status codes from 2xx to 5xx. That last part is the most useful: you see bots hitting 403s and 500s on pages you want cited.
Each crawler can be allowed or blocked, which writes a WAF custom rule. Pay per crawl, where crawlers are charged per request with a 402 response, is a closed beta.
The limits follow from where it lives. It sees only Cloudflare traffic. On the Free plan, crawlers are identified by user agent alone, so only self-identifying bots show up. Referral metrics and custom block responses need a paid plan; Cloudflare plans read on 20 October 2026 were Free $0, Pro $20, Business $200 a month billed annually ($250 monthly) and custom Contract.
How Known Agents works
Facts were read on knownagents.com/pricing, /about and /llms.txt on 20 October 2026. Known Agents is the company formerly called Dark Visitors, founded in New York by Gavin King. It started in 2023 as a side project listing the eight AI bots then known and now runs a public directory of more than 2,000 agents. It says more than 5,000 websites use it.
Four products share one event stream. Agent Analytics shows bots by page with spike alerts, error analysis, robots.txt violations and spoof verification. AI Chat Referral Tracking counts human visits from ChatGPT, Claude, Gemini, Copilot, Perplexity and others. Automatic Robots.txt adds rules as new agents appear. An Agent Identification API verifies agents by IP or message signature.
Pricing: Free with 1 project and 100K events, Startup at $29 a month (3 projects, 1M events), Growth at $299 (10 projects, 10M events), Enterprise custom. Every API call counts as an event, and Known Agents suggests budgeting about twice your human visits. Some routes depend on your own stack: Cloudflare Logpush needs Cloudflare Enterprise and the Vercel log drain needs Vercel Pro or Enterprise.
Its citation counts are estimates inferred from agents fetching your pages, as its own llms.txt says.
Who should pick which
Pick Cloudflare AI Crawl Control if your site is already behind Cloudflare. Turn it on today. It costs nothing and the status-code view alone will find broken pages.
Pick Known Agents if you run on Fastly, Akamai, CloudFront or Google Cloud, or across several stacks, or if you want robots.txt rules that update themselves. The free tier is enough for a small site.
You can run both on a Cloudflare site: Cloudflare for allow and block rules, Known Agents for AI referral counts and robots.txt automation. Check which Known Agents route your Cloudflare plan supports first, since Logpush needs Enterprise.
The Promptwatch option
Both tools stop at the fetch. They can tell you GPTBot requested /pricing and got a 200. They cannot tell you whether ChatGPT then cited /pricing in an answer to a buyer, because neither asks ChatGPT anything.
Promptwatch's Agent Analytics reads the same kind of logs, Cloudflare included, and joins them to the other half. Prompt tracking records what the engines answer. Citation analytics shows which of your pages they cite, at page, domain, Reddit and YouTube level. The crawl-to-citation path connects the two, and visitor analytics ties the resulting AI referrals to conversions. Logs cover ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther and the Meta AI crawler.
Crawler logs start on Professional at $245 a month (25M logs, 150 prompts, 1M visitor events). Business at $579 raises that to 100M logs. Keep Cloudflare's blocking controls; Promptwatch does not replace a WAF.
Verdict
Cloudflare AI Crawl Control: free, already running, Cloudflare-only. Known Agents: broader source coverage, better controls, estimated citations. Both are good at measuring fetches. For knowing what those fetches earned, the answer is Promptwatch, because citations come from tracked answers rather than inference.
FAQ
Is Cloudflare AI Crawl Control free?
Yes, on every Cloudflare plan including Free. Referral metrics and custom block responses need a paid plan.
Is Known Agents the same as Dark Visitors?
Yes. Dark Visitors renamed itself Known Agents to reflect a broader product.
Does either tool show whether ChatGPT cites my site?
No. Cloudflare shows crawler requests and, on paid plans, referrals. Known Agents estimates citations from agent fetches. Neither runs prompts.
Connect your crawler logs at promptwatch.com.