Agentic readiness check: can AI agents use your site?

Plenty of teams spend months on content for AI answers and never check the basic question: can the machines even get in? A site can have excellent pages and still be unreachable to the crawlers that feed AI engines, because of an old robots.txt rule, a bot-protection setting or a firewall someone switched on and forgot.

This guide covers how to test that, what the newer agent standards are and which of them matter, and a free check that runs the tests in one go.

Start with access, not standards

The first layer is plain crawler access. If an AI bot cannot fetch your page, nothing else you do matters for that page.

Check these in order:

robots.txt. Open yourdomain.com/robots.txt. Look for Disallow: / under a wildcard user agent, and for rules naming AI bots such as GPTBot, ClaudeBot or PerplexityBot. Rules are written per user agent, so a block on one bot does not block another.
Bot protection and firewall. A web application firewall or CDN setting that challenges unknown automation can block legitimate crawlers while still letting robots.txt look open.
Rendering. If key content is injected by JavaScript after load, a bot that does not run scripts sees an empty shell.
Status codes. Pages that redirect in loops or return errors to bots waste the visit.

Each bot has a purpose. Some collect training data, some power search, and some fetch a page when a user asks. Blocking one category is a legitimate choice, but it should be a choice, not an accident. Our guide on technical optimisation for AI crawlers covers the access basics in more depth.

The newer agent standards

Beyond crawling, a set of conventions has appeared for sites that want agents to do more than read. Here is what each is and what to expect from it:

llms.txt. A plain-text file that describes your site for language models. Whether any engine uses it is not established, so treat it as cheap documentation rather than a ranking lever. Our post on whether you need an llms.txt file goes through the evidence.
MCP (Model Context Protocol). A way for an agent to call tools exposed by a service. A site or product can publish an MCP server so agents can query or act on it.
A2A (agent to agent). Agent cards that describe what an agent can do so other agents can discover and talk to it.
OAuth discovery. Metadata that lets an agent find out how to authenticate with your service.
Content Signals. Directives that state how your content may be used, for example for search, for AI input or for training.

Most content sites do not need MCP or A2A. They matter if you run a product or service that agents might act on. Content Signals and robots.txt matter for everyone, because they express your policy.

Do not overclaim. Having these files does not guarantee any engine will use them. They are signals of readiness, not promises of visibility.

Doing the check by hand

You can test most of this with a browser and a terminal:

Fetch robots.txt and read the rules for each AI user agent.
Request /llms.txt and see if it exists and loads.
Request the well-known paths for MCP server cards, agent cards and OAuth metadata, if you publish them.
Request a page with an AI crawler user agent string and compare the response to a normal browser.

It works, but it is fiddly, and you will probably forget a path. Interpreting the results takes some knowledge, too.

Where Promptwatch fits

Promptwatch offers a free Agentic Readiness Check that runs these probes together. Per the September 8, 2026 changelog entry, you enter a domain and the tool shows which AI crawlers can reach it and which agent standards it supports, including llms.txt, MCP, A2A, OAuth discovery and Content Signals. Every result gets a shareable page, so you can send the outcome to a developer or a client without screenshots.

The check reads robots.txt against nine AI crawlers, and probes for llms.txt, MCP server cards, A2A agent cards, OAuth discovery and Content Signals. It is free, and no plan is needed. There is also a companion generator for your llms.txt, and a free AI Visibility Report that scans your brand without a login.

Why use it first? Because it is the cheapest possible diagnosis. In a few minutes you learn whether the entire problem is access. If it is, you have saved yourself from analysing content that no crawler ever reads.

From a one-time check to ongoing evidence

A readiness check is a snapshot of what is permitted. It does not show what actually happens. For that you need crawler logs, which show which bots arrive, which pages they fetch and whether they hit errors. Promptwatch's Agent Analytics provides IP-verified AI crawler logs and a crawl-to-citation path, which lets you move from "bots are allowed" to "bots visited and the page got cited". Crawler logs start on the Professional plan at $245 a month.

So the sensible order is:

Run the free check and fix anything it flags.
Wait a week or two and look at real crawler traffic, if you have the logs.
Look at whether pages that bots reach get cited.
Only then spend effort on content changes.

If your host does not let you see server logs, our post on crawler logs on Shopify, Webflow and WP Engine shows how to get them through the edge.

What to fix first

If the check turns up problems, use this order:

Accidental blocks. An overbroad disallow or a firewall rule. Fix these first, since they remove everything.
Policy decisions. Decide deliberately which bots you allow and why. A publisher may allow search bots and restrict training bots. Write the decision down.
Content Signals. Add them if you want to express usage preferences clearly.
llms.txt. Optional and cheap. Publish it if it costs you little, and do not expect results.
MCP, A2A and OAuth. Only if you run something agents can act on.

Run the check again after each change and compare. A shareable result page makes that easy, because you can keep the earlier ones as a record.

The takeaway

Readiness is the floor, not the ceiling. If agents cannot reach you, no content effort helps. If they can, the next questions are about what they fetch and what they cite. Start with the free check, fix access, and measure the rest. The best tools for checking AI crawler accessibility roundup lists alternatives if you want a second opinion.