How to Detect AI Agents Using or Citing My Content
Fetches, citations and clicks are three AI signals with three blind spots. The signal table, the user agents and IP lists to check, and seven steps to join them per URL.
Three signals. Three places to look. Most teams check one and assume the other two.
| Signal | Where to look | What it proves | Blind spot |
|---|---|---|---|
| 1. Fetch | Server or CDN logs, user agent plus IP list | A bot requested the URL and got a response | Not a citation. Training fetches don't feed search at all |
| 2. Citation | Answers to prompts you run and record | The engine used your URL as a source, for that prompt, that day | Prompts you don't track are invisible |
| 3. Click | Analytics referrers, utm_source=chatgpt.com | A human arrived from an AI answer | Citations with no click leave nothing; no prompt attached |
Two rules follow from the table. A crawl is not a citation. And a citation without a click doesn't exist in your analytics.
Signal 1: fetches
Step 1. Filter logs by user agent
The names to search for, taken from each vendor's own docs:
| User agent | Operator | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search |
| GPTBot | OpenAI | Training |
| ChatGPT-User | OpenAI | Visits a user triggered from ChatGPT or a Custom GPT |
| ClaudeBot | Anthropic | Training |
| Claude-SearchBot | Anthropic | Indexing for Claude search quality |
| Claude-User | Anthropic | Fetches when a user asks Claude |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search; not used for training |
| Perplexity-User | Perplexity | Fetches a user requested |
| GoogleOther | Generic crawler for Google product-team fetches |
Sources: OpenAI, Anthropic, Perplexity, Google.
Missing on purpose: Google-Extended. It has no user agent string. It's a robots.txt token for Gemini training and grounding and doesn't affect Google Search inclusion. Nothing to grep. GoogleOther is in the table, but it isn't AI-specific, so treat its hits as a weak signal.
Step 2. Verify against IP lists
User agents can be spoofed. Google says so and points to its published IP ranges and reverse DNS for verification. The lists to check:
- OpenAI: openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json
- Perplexity: perplexity.com/perplexitybot.json, perplexity.com/perplexity-user.json
Perplexity recommends matching user agent and IP together in WAF rules. Anthropic warns that IP blocking may not work reliably. Report unverified requests in their own bucket. Don't add them to the bot counts.
Step 3. Split by purpose
- Training (GPTBot, ClaudeBot): tells you nothing about AI search visibility.
- Search indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot): the precondition for being cited in those engines.
- User-triggered (ChatGPT-User, Claude-User, Perplexity-User): a live conversation pulled your page. This is the closest log signal to an answer. It still doesn't show whether the answer cited you.
Robots.txt behaves differently per group. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores them. Anthropic says its three bots respect robots.txt and support Crawl-delay. Robots changes take about 24 hours to register for OpenAI search and up to 24 hours at Perplexity.
Step 4. Check status codes
A 403 or 5xx served to OAI-SearchBot is an access problem, not a content problem. Fix it before you rewrite anything.
Signal 2: citations
Step 5. Run prompts and record cited URLs
No server log holds this. Pick the prompts buyers ask, run them on a fixed schedule across the engines that matter to you, and store every cited URL. Page level, not just domain. Third-party pages about you count too: a Reddit thread or YouTube video cited for your category is part of your citation footprint.
Signal 3: clicks
Step 6. Segment AI referrals
OpenAI's publisher FAQ says ChatGPT adds utm_source=chatgpt.com to referral URLs. Filter on it. Group the other assistant referrers you see into one AI segment, then attach conversions. Sessions alone don't tell you much.
Step 7. Join the three per URL
| Fetched | Cited | Clicked | Read it as |
|---|---|---|---|
| No | No | No | Access problem first. Check steps 2 to 4 |
| Yes | No | No | Reachable but not chosen. Content or relevance gap |
| Yes | Yes | No | Visible without traffic. The answer may settle the question without a click |
| Yes | Yes | Yes | Working. Protect the page |
| No recent fetch | Yes | Any | Possibly from earlier retrieval. Don't assume a causal path |
Without the join, each signal on its own will mislead you.
Tool verdicts
1. Promptwatch. Covers all three signals. Agent Analytics ingests real-time crawler logs via Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN or custom HTTP, with error tracking and a crawl-to-citation path, which is step 7 done for you. Citation analytics works at page and domain level, with Reddit and YouTube citations reported separately. Visitor analytics tracks AI-referred visits and conversions through a script or a GTM template. Numbers: Professional is $245/mo for 150 prompts, 18,000 responses, 1M visitor events and 25M crawler logs. Essential is $95/mo; Business is $579/mo with 100M crawler logs. Verdict: the only tool here that answers "crawled, cited, clicked, converted" on one URL.
2. Cloudflare AI Crawl Control. Signal 1 only. Free on every Cloudflare plan. Requests by operator, status codes, top paths, allow or block per crawler. Detection is user agent only on Cloudflare Free. Referral metrics need a paid plan, from $20 on Pro. No prompts, no citations. Verdict: a free first look at step 1 for Cloudflare sites.
3. Known Agents. Signals 1 and 3. Broad log sources, spoof verification, AI chat referral tracking. Its citation counts are estimated from agent fetches, per its own llms.txt, so signal 2 is inferred rather than observed. Free plan available. Verdict: strong log tool, not a citation tracker.
For more log-only options, see our ranking of standalone AI crawler analytics tools. Full leaderboard: /#tools. Start a trial at promptwatch.com.