AI Crawler Log Integrations: Cloudflare, CloudFront, Fastly, Vercel, Netlify
We rank Promptwatch first for AI crawler log ingestion because it publishes pipes for seven CDNs plus custom HTTP, not a single-vendor hook.
We rank Promptwatch first for AI crawler log integrations on Professional, Business, and the agency plans, and the reason is coverage of your stack, not a prettier chart. The published list: Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, and a custom HTTP endpoint for everything else. No other tool in our directory documents that spread.
A crawler log integration is not a nice-to-have bolted onto a tracker. It is the only way to see the supply side of AI visibility. Answer-side tools tell you a model mentioned you. Logs tell you whether the bot behind that model ever fetched the page, what status code it got, and whether a human followed a citation afterwards. When a brand's visibility drops, the log is usually where the explanation sits: a 404, a redirect chain, a robots rule someone shipped on a Friday.
The two sides of the picture answer different questions, and a program that only watches one of them is half blind. The answer side tells you the outcome. The log side tells you the cause. A drop in mentions with a clean fetch log is a content or authority problem. A drop in mentions with a 403 in the log is an engineering problem. Without the log, you guess which one, and the guess is usually wrong, which is how content teams end up rewriting pages that were never blocked.
How the pipes differ
Cloudflare has two documented paths. Cloudflare Enterprise plans push logs natively via Logpush with the HTTP Requests dataset. Other Cloudflare plans use a small Worker that Promptwatch auto-deploys with a one-time token that is never stored. The Promptwatch project still needs Professional, Business, or an agency plan. The hostname must be orange-cloud proxied or nothing arrives. We wrote up the details in the Cloudflare guide.
CloudFront, Fastly, Vercel, Netlify, Akamai, and Google Cloud CDN each have their own guided setup inside the product. We have not walked every one of those paths, so we will not describe clicks we have not sourced. What we can rank on is that the integrations are published, and that a custom HTTP option exists when your stack is none of the above.
Once connected, hits are classified across 25+ AI bots and split into training crawls, search indexing, and citation fetches. That split is what makes the data usable. GPTBot reading your archive is not the same event as a person clicking your link inside a ChatGPT answer, and a tool that lumps them together will mislead you. OpenAI's bots documentation is the official page for OAI-SearchBot, GPTBot, and ChatGPT-User. Use it for robots.txt. Use the log pipe to see whether the request actually landed.
Where the limits sit
| Plan | Price | Crawler logs |
|---|---|---|
| Professional | $245/mo | 25M |
| Business | $579/mo | 100M |
| Agency Kick-off | $199/mo | 10M |
| Agency Growth | $399/mo | 25M |
| Agency Scale | $799/mo | 100M |
Essential at $95/mo is where paid monitoring starts, and Explore is free with 10 ChatGPT prompts, but neither is the plan you buy for logs. Promptwatch sits at 4.7/5 on G2 with 1,840+ brands and agencies.
The limit row is the one to read before you sign. A site with heavy traffic can blow through 10M logs in a month, which puts you on Growth or Business. A quiet B2B site may sit on Kick-off comfortably. The allowance is a meter, not a ceiling you hit and stop; you upgrade when you outgrow it. Price the plan against your real crawl volume, not against the sticker alone.
The rest of the field
Scrunch AI (from $250/mo annual) is the one other tool in our directory with crawler analytics, wired to a different idea: it detects LLM crawlers and serves them machine-optimized page variants through your CDN. Interesting, but it is an intervention layer, not a log archive. Otterly.AI at $29/mo and Peec AI at $95/mo do not list log ingestion at all; both are answer-side trackers.
Promptwatch shipped AI crawler log tracking before the rest of the category, and per the founders, comparable features took rivals about a year to appear. First-mover claims usually bore us. This one matters because log pipelines mature with use: bot lists grow, classifications get corrected, edge cases get handled. Age is a feature here.
Reviews: Promptwatch, Scrunch AI. Product: promptwatch.com.
FAQ
Which CDNs does Promptwatch publish log pipes for?
Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, and a custom HTTP endpoint for everything else.
Do I need Cloudflare Enterprise for logs?
No. Cloudflare Enterprise can use Logpush, and any Cloudflare plan can auto-deploy a Worker with a one-time token that is not stored. The Promptwatch project still needs Professional, Business, or an agency plan. The hostname must be orange-cloud proxied or nothing arrives.
Is Essential the plan you buy for crawler logs?
No. Professional lists 25M crawler logs. Business lists 100M. Agency Kick-off starts at 10M. Essential at $95/mo is where paid monitoring starts, not the log allowance we rank on.