Reading the Crawl-to-Citation Path Page by Page
How to use a crawl-to-citation view to sort pages into four groups: crawled and cited, crawled only, cited only, and invisible, and what to do with each.
If you run a site long enough, you learn that two numbers rarely agree: how often bots visit a page, and how often AI answers cite it. The interesting work happens in the gap. A crawl-to-citation view puts both numbers side by side per page, and a simple four-box sort turns it into a to-do list.
Promptwatch's Agent Analytics includes this: a crawl-to-citation path with a citation rate per page, built on real-time, IP-verified crawler logs. Here is how to use that kind of view whatever your tooling.
The four boxes
Take your pages and place each one in a box based on two questions. Was it crawled by AI bots recently? Was it cited in tracked answers?
| Cited | Not cited | |
|---|---|---|
| Crawled | Working | Content or relevance problem |
| Not crawled | Cached or stale win | Access or discovery problem |
Box 1: crawled and cited
These pages are doing the job. Do three things: protect them, learn from them, and watch for decay. Protect means no accidental edits that remove the sections that earned the citations. Learn means looking at the structure, the specificity and the question the page answers, then repeating it. Watch means adding them to a Page Tracker so a dip is visible.
Box 2: crawled, not cited
The most interesting group. The bots found the page, fetched it and still did not use it. That points away from access and toward content or fit. Typical causes:
- The page answers a question nobody in your tracked panel asks.
- It covers the topic in generalities where a competitor offers specifics.
- A better source for the same question already exists, such as a forum thread or a comparison on a third-party site.
- The title and opening do not signal what the page can answer.
The next step is to check query fan-outs for the prompts closest to the page's topic. Promptwatch shows what models searched for per prompt. If the sub-questions do not match your page, rewrite toward them. Content gap analysis with coverage scores helps you spot topics where you have a page but thin coverage.
Box 3: cited, not crawled recently
This sounds odd until you remember that systems cache. A page can be cited from an earlier fetch. The risk is that the page becomes stale in the cache, or that a recent change broke access and you have not seen the effect yet. Treat this box as a watch list. Look at the date of the last successful crawl and check for errors since.
Box 4: not crawled, not cited
Either the page does not matter, or no bot can reach it. Sort the two. Pages that should matter but sit here need a diagnostic: is it in the sitemap, linked internally, allowed by robots.txt, and answered with a 200 at the edge? Promptwatch's weekly site audit covers 21 checks, a health score and crawlability checks, which gives you a baseline for these pages.
Doing it in practice
- Export or filter your top 50 pages by business value, not by traffic.
- Record crawl and citation status for each over the same date range.
- Place them in the four boxes and count.
- Pick one box for the week. Most teams should start with box 2, because the fix is in your hands and the effect shows up fast.
Keep the date range consistent. A 7-day window and a 90-day window will place the same page in different boxes.
Reading citation rate
A citation rate per page, in Promptwatch's crawl-to-citation view, relates citations to crawls. A page with many crawls and a tiny rate is a box 2 page. A page with few crawls and a high rate is efficient but possibly under-fetched, which is a prompt to check access and internal linking so the good page gets more attention. Use rates as a ranking tool for attention, not as a grade.
What the view cannot tell you
It cannot tell you the intent behind a crawl. Some bots fetch broadly, others fetch because of a specific request. It cannot tell you that a citation produced a visit either. For that you need AI-referred traffic data: Promptwatch's visitor analytics track AI-referred sessions and conversions from ChatGPT, Perplexity, Gemini, Claude, Copilot and others. The 360 Insights view joins crawler logs, citation clicks and AI-referred visits per page, which gets you to the end of the chain.
Mistakes to avoid
- Treating all crawls equally. A crawl by a broad collector and a crawl by an on-demand fetcher mean different things.
- Ignoring errors. A page with some 200s and many 5xx responses lands in the wrong box if you only count visits.
- Fixing box 2 with more links. If the content does not answer the sub-question, more links to it will not help.
Next step
Run the sort on your twenty most valuable URLs and note which box is biggest. That box is your quarter's theme. Promptwatch has the crawl-to-citation path, error tracking, Page Tracker and 360 Insights in one place, so you can build the four boxes from a single screen rather than stitching a CDN export to a citation spreadsheet. Professional is where crawler logs begin on brand plans, and agency plans include them from Kick-off.