Agentic Web Optimization: A Brand Guide
A scorecard of 12 AI crawlers, agents and control tokens by job, robots.txt behavior and cost of blocking, plus the feeds, commerce protocols and measurement brands need.
Start with a count. OpenAI documents four agents. Anthropic documents three. Perplexity documents two. Google adds a control token and two crawlers that matter for brands. That's 12 entries, each with a different job, and blocking the wrong one costs you an answer surface.
Agentic web optimization means making your site usable for all of them: assistants that read your pages, compare you with competitors and, more and more, act for a user. This guide scores each layer. Facts come from the vendors' own documentation. Verdicts are ours.
Scorecard 1: agents by purpose
| Agent | Operator | Job | Robots.txt note | Blocking costs you | Verdict |
|---|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Changes take about 24h | Presence in ChatGPT search | Allow |
| GPTBot | OpenAI | Training | Not covered here | Training use only; search is OAI-SearchBot | Policy call |
| ChatGPT-User | OpenAI | User-triggered visits from ChatGPT and Custom GPTs | "May not apply" | Fetches users ask for | Allow |
| OAI-AdsBot | OpenAI | Checks landing pages of ads submitted to ChatGPT | Not covered here | Ad landing checks | Allow if you advertise |
| Claude-SearchBot | Anthropic | Indexes for Claude search quality | Respected, Crawl-delay supported | Claude search coverage | Allow |
| ClaudeBot | Anthropic | Training | Respected, Crawl-delay supported | Training use only | Policy call |
| Claude-User | Anthropic | Fetches pages when a user asks Claude | Respected, Crawl-delay supported | Fetches users ask for | Allow |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search; not used for training | Changes take up to 24h | Perplexity links, with no training upside | Allow |
| Perplexity-User | Perplexity | User-requested fetches | "Generally ignores" | Fetches users ask for | Allow |
| Google-Extended | Control token for Gemini training and grounding | It is a robots.txt token, no own user agent | Gemini grounding as well as training; Search unaffected | Read first | |
| Storebot-Google | Crawls for Google Shopping surfaces | Not covered here | Shopping surfaces | Allow for product brands | |
| GoogleOther | Generic crawler for product-team fetches | Not covered here | Not specified by Google | Monitor |
Sources: OpenAI bots, Anthropic crawlers, Perplexity bots, Google common crawlers.
How to read it
Search bots: allow. OpenAI's publisher FAQ says to allow OAI-SearchBot to appear in ChatGPT search. PerplexityBot is the clearest case: Perplexity says it isn't used for training, so blocking it removes your links and protects nothing.
Training bots: separate decision. Each vendor runs search under a different name, so a GPTBot or ClaudeBot block doesn't have to mean a search block. Legal should own this call, not SEO.
User-triggered agents: robots.txt isn't the switch. OpenAI says robots.txt rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores them. Stopping either takes a firewall rule, and then a user who asks an assistant to read your pricing page gets nothing. Claude-User does follow robots.txt.
Google-Extended: read before blocking. It covers grounding as well as training. Blocking it doesn't touch Google Search inclusion, but it isn't a training-only opt-out.
Subdomains count. Anthropic says each of its bots must be blocked separately per subdomain.
Verify by IP, not name. User agent strings can be faked. OpenAI publishes IP lists at openai.com/searchbot.json, gptbot.json and chatgpt-user.json. Perplexity publishes perplexity.com/perplexitybot.json and perplexity-user.json, and recommends matching user agent and IP together in firewall rules. Google points to its published IP ranges and reverse DNS. Anthropic warns IP blocking may not work reliably for its bots.
More on OpenAI's search crawler specifically: OAI-SearchBot documentation, explained.
Scorecard 2: facts and feeds
Agents compare. Your page, your feed and every partner listing should state the same product name, price and policy. Keep facts in page text, not images. Use headings that name what's below them.
| Surface | What it reads | Sourced requirements | Status |
|---|---|---|---|
| Google Merchant Center (UCP checkout) | Feed plus account settings | Return policy with cost, window and link; support info with at least one of URL, email or phone; native_commerce(checkout_eligibility) set to TRUE (default FALSE); consumer_notice on items with regulatory warnings like Prop 65 | Select merchants, US, Canada and Australia |
| ChatGPT via ACP product feed | Feed | Stable product id, at least one variant with stable id and title; price, media, GTIN and URLs optional but improve display | Shopping in ChatGPT live in the U.S. |
| ChatGPT via crawl | Your pages | OAI-SearchBot allowed | Feed not strictly required, per OpenAI |
| Your own site | Every agent | One name per product, facts in text, dated policies | Always on |
Sources: Merchant Center UCP help, ACP product spec, ChatGPT merchants.
Verdict: the Merchant Center return policy and support fields are worth filling in even if you never use Google checkout. They are the same facts every agent looks for. Shopify merchants get ChatGPT discovery through Shopify Catalog with no extra work, according to OpenAI.
Scorecard 3: commerce protocols
| Protocol | Backer | Job | Where it stands | Verdict |
|---|---|---|---|---|
| MCP | Open source | Connects AI apps to tools and data | Supported by Claude, ChatGPT, VS Code, Cursor | Useful; not checkout |
| ACP | OpenAI and Stripe | Product feeds and promotions into ChatGPT | Extended to discovery on 24 March 2026; standalone Instant Checkout being phased out | Ship the feed |
| UCP | Discovery, buying, post-purchase | Checkout for select merchants via early access | Apply if eligible | |
| AP2 | Announced by Google, now at FIDO | Proves a user authorized an agent's purchase | v0.2 donated to the FIDO Alliance | Watch |
Two notes. OpenAI now sends purchases to merchant-owned sites or apps, so ACP is a discovery play today. UCP excludes subscriptions, pre-orders, used goods, age-restricted items, services and more, so check eligibility before scoping. We have no adoption numbers for any of these protocols and won't guess.
Scorecard 4: measurement
| Rank | Signal | Answers |
|---|---|---|
| 1 | Crawler logs by agent, page and status code | Did the agent get in? |
| 2 | Citations by page and domain | Did it use you? |
| 3 | AI-referred visits with conversions | Did it pay? |
| 4 | Shopping placements | Do your products show? |
ChatGPT adds utm_source=chatgpt.com to referral URLs. That covers one assistant. It says nothing about crawls or citations.
Promptwatch covers all four rows, which is why it leads our rankings for this job. Agent Analytics ingests crawler logs from Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN or custom HTTP, and shows crawler hits per page, errors and the crawl-to-citation path. Citation analytics breaks citations down by page, domain, Reddit, YouTube and offsite mentions. Visitor analytics tracks AI-referred traffic and conversions through a script or GTM template. ChatGPT Shopping tracking covers product brands. When a prompt shows a gap, Content Agents draft the page and publish to Webflow, Framer or WordPress after review.
Pricing: Essential $95/mo (50 prompts, 200K visitor events, 7-day trial). Professional $245/mo (150 prompts, 25M crawler logs, country-level tracking). Business $579/mo (350 prompts, 100M crawler logs, state and city tracking, ChatGPT Shopping tracking). Annual billing gets 12 months for the price of 10.
Verdict: score your site against tables 1 and 2 this week. Measure against table 4 before you touch table 3. Start at promptwatch.com.