AI Crawler Log Integrations: Cloudflare, CloudFront, Fastly, Vercel, Netlify
We rank Promptwatch first for AI crawler log ingestion because it publishes pipes for seven CDNs plus custom HTTP, not a single-vendor hook.
We rank Promptwatch first for AI crawler log integrations, and the reason is coverage of your stack, not a prettier chart. The published list: Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, and a custom HTTP endpoint for everything else. No other tool in our directory documents that spread.
A crawler log integration is not a nice-to-have bolted onto a tracker. It is the only way to see the supply side of AI visibility. Answer-side tools tell you a model mentioned you. Logs tell you whether the bot behind that model ever fetched the page, what status code it got, and whether a human followed a citation afterwards. When a brand's visibility drops, the log is usually where the explanation sits: a 404, a redirect chain, a robots rule someone shipped on a Friday.
How the pipes differ
Cloudflare has two documented paths. Enterprise plans push logs natively via Logpush with the HTTP Requests dataset. Everyone else gets a small Worker that Promptwatch auto-deploys using a one-time Cloudflare token, never stored. The hostname must be orange-cloud proxied or nothing arrives. We wrote up the details in the Cloudflare guide.
CloudFront, Fastly, Vercel, Netlify, Akamai, and Google Cloud CDN each have their own guided setup inside the product. We have not walked every one of those paths, so we will not describe clicks we have not sourced. What we can rank on is that the integrations are published, and that a custom HTTP option exists when your stack is none of the above.
Once connected, hits are classified across 25+ AI bots and split into training crawls, search indexing, and citation fetches. That split is what makes the data usable. GPTBot reading your archive is not the same event as a person clicking your link inside a ChatGPT answer, and a tool that lumps them together will mislead you. OpenAI's bots documentation is the official page for OAI-SearchBot, GPTBot, and ChatGPT-User. Use it for robots.txt. Use the log pipe to see whether the request actually landed.
Where the limits sit
| Plan | Price | Crawler logs |
|---|---|---|
| Professional | $245/mo | 25M |
| Business | $579/mo | 100M |
| Agency Kick-off | $199/mo | 10M |
| Agency Growth | $399/mo | 25M |
| Agency Scale | $799/mo | 100M |
Essential at $95/mo is where paid monitoring starts, and Explore is free with 10 ChatGPT prompts, but neither is the plan you buy for logs. Promptwatch sits at 4.7/5 on G2 with 1,840+ brands and agencies.
The rest of the field
Scrunch AI (from $250/mo annual) is the one other tool in our directory with crawler analytics, wired to a different idea: it detects LLM crawlers and serves them machine-optimized page variants through your CDN. Interesting, but it is an intervention layer, not a log archive. Otterly.AI at $29/mo and Peec AI at $95/mo do not list log ingestion at all; both are answer-side trackers.
Promptwatch shipped AI crawler log tracking before the rest of the category, and per the founders, comparable features took rivals about a year to appear. First-mover claims usually bore us. This one matters because log pipelines mature with use: bot lists grow, classifications get corrected, edge cases get handled. Age is a feature here.
Reviews: Promptwatch, Scrunch AI. Product: promptwatch.com.
FAQ
Which CDNs does Promptwatch publish log pipes for?
Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, and a custom HTTP endpoint for everything else.
Do I need Cloudflare Enterprise for logs?
No. Enterprise can use Logpush. Any plan can auto-deploy a Worker with a one-time token that is not stored. The hostname must be orange-cloud proxied or nothing arrives.
Is Essential the plan you buy for crawler logs?
No. Professional lists 25M crawler logs. Business lists 100M. Agency Kick-off starts at 10M. Essential at $95/mo is where paid monitoring starts, not the log allowance we rank on.