Robots.txt for AI Search: Allow the Bots That Can Cite You
A practical robots.txt and firewall checklist for AI search: what to allow, what to test, and how to confirm in crawler logs that the change worked.
Robots.txt is a short file with a long reach. One careless line can cut your site out of AI answers, and nothing will tell you. The change just shows up weeks later as a vague drop in visibility. This is the checklist we use when we audit a site's AI crawler access.
First, decide what you want
Before touching the file, answer one question: do you want AI products to be able to show and cite your pages? For most businesses selling something, yes. For a publisher with licensing concerns, the answer may be mixed. The file should encode a decision, not an accident.
Write down the policy per bot. The real names, per the list Promptwatch tracks, include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher and MistralAI-User. Promptwatch's free AI Bots directory lists 440+ crawlers if you meet one you do not recognize.
The checklist
1. Read the current file
Open yourdomain.com/robots.txt. Look for these patterns:
- A blanket
Disallow: /underUser-agent: *. - Agent-specific blocks added during a past "protect our content" scare.
- Rules copied from a template nobody reviewed.
2. Match rules to roles
Think in terms of jobs. Broad collection crawlers, index-building bots and on-demand fetchers are separate, and a rule that is right for one may be wrong for another. If you want to appear in answers, the search and user-fetch bots are the ones to leave open.
3. Remember what robots.txt cannot do
It is a request, not a lock. Compliant bots follow it; others do not. It also says nothing about your CDN, WAF or bot-management rules. A site can have a perfect robots.txt and still return 403 to every AI bot at the edge.
4. Check the firewall layer
Look at rate limits, "bot fight" modes and managed rulesets. Ask your infrastructure team what happens to unknown or non-browser user agents. If the answer is "challenge them," your allow rule in robots.txt is decorative.
5. Publish a sitemap and keep it fresh
A sitemap tells crawlers what exists. Promptwatch's weekly site audit includes crawlability checks, and a sitemap progress view in its tooling can show how far through your sitemap crawlers have gotten.
6. Consider llms.txt as a courtesy, not a fix
An llms.txt file gives language-model products a readable summary of your site. Promptwatch offers free llms.txt and robots.txt generators. Treat llms.txt as an extra signal, not a replacement for access. Do not expect it to rescue a blocked site.
7. Test with evidence, not hope
After any change, wait for traffic and read the logs. In Promptwatch, Agent Analytics shows real-time, IP-verified crawler hits with error tracking and failure explanations. You want to see that the search and user-fetch bots now receive 200 responses on your key pages.
A short example of reasoning
Suppose your logs show ChatGPT-User receiving 403s on your pricing page. Your robots.txt allows it. That means the block is at the edge: a WAF rule, a geo rule, or a rate limit. Fixing the robots.txt would change nothing. Without logs, you would have rewritten the wrong file twice.
Now suppose OAI-SearchBot fetches the page every few days with 200s and the page is never cited. Access is fine. Look at the content, its structure, and whether the prompts you track are actually answered by it. That is a different problem with a different owner.
Common mistakes
- Wildcard panic. Blocking every user agent containing "bot" removes Googlebot-adjacent and AI search agents together.
- Staging rules in production. A
Disallow: /that survived a launch. - Forgetting subdomains. Each host has its own robots.txt. Your docs or blog subdomain may behave differently from the main site.
- No review date. Bot names change. Put a quarterly reminder on the calendar.
What to do when you cannot change the edge
Some teams do not own their firewall. In that case, bring evidence to the people who do. A CSV export of failed bot requests with failure explanations is far more persuasive than a request that says "we think AI bots are blocked." Promptwatch supports CSV export from Agent Analytics for exactly this handoff.
Confirm the business outcome
Access is a means. The outcome is citations and visits. After fixing access, track the pages in Page Tracker and look at citation counts and AI click-throughs next week, then next month. Visitor analytics will show AI-referred sessions and conversions if you have the script, GTM template or WordPress plugin installed.
Recommendation
Make the bot policy a document, test it in logs, and keep a changelog of robots.txt edits next to your visibility chart. If you have no way to see which bots are hitting which pages, that is the first gap to close. Promptwatch provides IP-verified crawler logs, tracked bot names, error explanations and CSV export, plus the free generators for robots.txt and llms.txt. Connect your CDN, give it a week, and let the logs tell you whether your file does what you think it does.