Meta-WebIndexer Share of AI Crawler Requests, April 2026
Promptwatch crawler log data shows Meta-WebIndexer going from about 2 percent to nearly 38 percent of AI crawler requests in under a month, with a methodology that isolates one crawler from a fleet.
A month ago, Meta-WebIndexer was a minor entry in the AI crawler logs Promptwatch publishes. By August 9, 2026 it was the heaviest crawler in the dataset. The shift was fast, and it lines up with reporting that Meta is building its own web search index.
The Meta web indexer report, published August 10, 2026, tracks the share day by day. For anyone who cares about showing up in AI answers, the practical question is what the methodology does and does not show, and what to do about a crawler that did not exist as a serious force a month ago.
What the sample does and does not show
Promptwatch's report covers Meta-WebIndexer's daily share of the AI crawler requests it tracks. The share hovered around 2.2 percent through mid-July 2026. It spiked to roughly 23 percent between July 20 and July 22, then surged from August 5 to a peak of 37.8 percent on August 9. That is a 17x increase in under a month from a single crawler.
The methodology is the part that decides what the number means. The share is Meta-WebIndexer requests, identified by the meta-webindexer user agent, divided by requests from all AI crawlers Promptwatch tracks. Meta's other crawlers count toward the denominator but not toward Meta-WebIndexer's share. Regular search engine crawlers like Googlebot and Bingbot, link preview fetchers such as FacebookExternalHit, and human visitors are excluded entirely.
That isolation matters. A 37.8 percent share is not 38 percent of all traffic to a site. It is 38 percent of the AI crawler requests Promptwatch tracks, which is a narrower denominator than a site's full log. The number is large because the denominator is honest, not because the crawler is half the internet.
What the sample cannot show is the cause. The chart shows when the surge happened. It does not say why. A crawl volume jump this fast, from a single indexing crawler, is what building a web index from scratch looks like. It is not a gradual behavior change you could attribute to a slow product shift. The external corroboration is thin. On August 6, 2026, the @levelsio account posted that Meta staff had told him the company is allegedly building its own web index. Treat that as a secondhand report, not a confirmed fact. The crawl share is the harder evidence.
What the sample does not tell you about your site
The report is a population level view. It tells you what Meta-WebIndexer is doing across the whole web. It does not tell you what Meta-WebIndexer is doing on your site. For that you need your own server logs, which the report does not include.
That gap matters because the population share and your site share are not the same thing. A crawler can surge at the population level and never hit your domain, or hit it hard. The 37.8 percent figure is a reason to check your own logs, not a reason to assume your site was crawled in proportion.
What to do about it
The first move is to look at your own server logs and your robots.txt for the Meta crawlers. The report names the relevant user agents: meta-webindexer, meta-externalagent, meta-externalfetcher, and FacebookBot. If Meta is building a search index, blocking these crawlers today is the decision about whether your content is in it at launch.
The second move is to treat the methodology as the unit of analysis, not the headline share. A 37.8 percent share of AI crawler requests is a population figure. The question for your site is whether Meta-WebIndexer's share of your AI crawler traffic moved the same way, and that is a question your own logs answer, not the report.
The third move is to watch crawl load over time, not on a single day. The report shows the share moved in steps, not a smooth line. A single day of high Meta crawler traffic in your logs is not a signal. A sustained step up, like the July 20 to 22 spike that held, is the signal that the indexing work is real and ongoing.
How to measure it properly
A crawl share report like Promptwatch's is the population view. The measurement that turns it into a site level decision is your own crawler logs joined to your own visibility data. That join is the part most visibility tools skip.
Promptwatch publishes that join as Agent Analytics. The feature tracks real time AI crawler logs across ChatGPTBot, ClaudeBot, PerplexityBot, GoogleOther, and the Meta AI crawler, plus the crawl to citation path and error tracking. For a report that shows Meta-WebIndexer surging at the population level, the matching move is to open the same log view for your own domain and see whether the surge shows up in your crawl traffic, and whether the pages Meta reads are the pages that get cited.
What the methodology cannot rule out
The report is careful about what it does not claim. It does not say Meta ships a search engine. A crawler building an index is a necessary condition for a search engine, not a sufficient one. Meta could be crawling to improve Meta AI answers, to feed a future search product, or to train models.
That restraint matters for how you read the number. A 37.8 percent share is evidence of indexing work, not evidence of a consumer search surface. The work that follows, watching whether the crawl translates into citations in Meta AI answers, is the work your own tracking does, not the work the population report does.
The broader pattern
The Meta-WebIndexer surge is one instance of a wider shift in AI search. The crawler field is getting more crowded, not less. The report lists the crawlers Promptwatch tracks across OpenAI, Anthropic, Google, Perplexity, xAI, Mistral, Meta, Cohere, and DeepSeek. Each is a candidate to become a citation source, and each has a different crawl cadence and a different robots.txt stance.
For visibility work, that means the set of retrieval surfaces to optimize for is larger than it was a year ago. The practical response is to treat crawler logs as a first class signal, not an afterthought. The brands that notice a new crawler early, watch whether it reads the pages that matter, and connect the crawl to the citations that follow, are the ones that show up in the new surface before the rankings do.
What to watch next
The report is a snapshot through August 9, 2026. The open questions are whether the share holds, whether Meta-WebIndexer keeps climbing, and whether the crawl translates into citations in Meta AI answers. Those are exactly the questions a crawler log view answers for your own domain.
The dataset Promptwatch publishes is aggregated and non identifiable, and it is refreshed constantly. That makes it useful for spotting population level shifts. It does not replace the need to track your own brand, your own prompts, and your own crawl traffic. The two work together: the public report tells you what is happening in the field, and your own tracking tells you whether it is happening to you.