June 23, 2026 8 min read

AI crawlers and AEO: what to allow, block and monitor

A practical guide to AI crawler access for AEO: separate training bots from search bots, avoid accidental blocks and verify whether access becomes citations.

AEO
AI Crawlers
Robots.txt
Technical SEO

Editorial illustration of AI crawlers passing through access rules toward search retrieval, answer citations and model training systems

AI crawler access is an AEO control, not a minor technical setting: if the right search and retrieval bots cannot reach your useful pages, those pages are less likely to be cited by answer engines. The practical rule is to separate training access from answer access, document the decision in robots.txt and WAF rules, and then measure whether crawl eligibility turns into citations, mentions and recommendations.

This is where many teams make the expensive mistake. They either allow every bot without knowing what each one does, or they block every AI crawler and accidentally remove their best pages from ChatGPT search, Claude search, Perplexity or other answer surfaces. AEO does not require blind openness. It requires a clear access policy.

The useful distinction: training, search and user fetch

An AI crawler is not one thing. The same company may operate separate agents for model training, search indexing and user-initiated fetching. Those jobs have different business implications. A training bot may collect public pages for future model improvement. A search bot may index or retrieve pages so an answer engine can cite them. A user fetch agent may visit a URL because a person explicitly asked the assistant to read or use that page.

For AEO, the safest default is not 'allow AI' or 'block AI'. The safest default is 'allow retrieval you want measured, block training only if that is a deliberate content-policy decision'.

OpenAI illustrates the split clearly. Its documentation distinguishes OAI-SearchBot for search-related crawling, GPTBot for content that may be used in foundation-model training, and ChatGPT-User for user-triggered actions. Anthropic uses a similar pattern with ClaudeBot, Claude-SearchBot and Claude-User. Perplexity's public crawler documentation presents PerplexityBot as a bot for surfacing and linking websites in Perplexity search results, not for foundation-model training. Google is different again: Google-Extended is a robots.txt product token, not a separate HTTP user agent, and Google says it does not affect inclusion or ranking in Google Search.

A practical crawler matrix for AEO teams

OpenAI search visibility: allow OAI-SearchBot on pages that should be eligible for ChatGPT search answers.
OpenAI training control: decide separately whether GPTBot may access content that could be used for future model training.
OpenAI user actions: understand that ChatGPT-User can be triggered by user actions and is not the same as automated search crawling.
Claude search visibility: allow Claude-SearchBot where Claude should be able to discover and index content for search responses.
Claude training control: decide separately whether ClaudeBot may access content for model-training datasets.
Claude user actions: treat Claude-User as a retrieval path for user-directed requests, not as a replacement for Claude-SearchBot.
Perplexity visibility: allow PerplexityBot and verify WAF access with Perplexity's published IP ranges if citations in Perplexity matter.
Google AI control: use Google-Extended as a specific policy token for Gemini-related training and grounding uses, while keeping Googlebot separate for Search.

This matrix is deliberately operational. It does not claim that allowing a bot guarantees a citation. It only keeps your eligible pages reachable. From there, answer engines still choose sources based on relevance, extractability, authority, freshness when relevant and the evidence they can safely cite.

Robots.txt is necessary, but not the whole access layer

Robots.txt is where you express intent to compliant crawlers. It is also the easiest file to audit publicly. For a brand that wants AEO visibility, the file should not rely on a vague wildcard rule and hope every AI system interprets it correctly. List the important AI user agents explicitly, then match those choices in the CDN, WAF, bot-management product and server logs.

That last part matters. A site can show a permissive robots.txt file and still block AI crawlers at the edge. Perplexity's own crawler documentation tells site owners using a WAF to allow both user-agent and IP conditions from its official endpoints. Cloudflare's bot documentation also makes clear that bot controls, managed robots.txt and verified-bot policies can sit outside the plain robots.txt file. In practice, AEO crawl audits have to inspect both layers: declared policy and actual access.

Three access strategies, and when each fits

1. Visibility-first access

This is the default for brands, agencies, SaaS companies, professional services firms and publishers that want their public expertise cited. Allow search and user-fetch bots. Allow training bots only if the organization is comfortable with that use. Monitor crawl load, cited URLs and conversion quality. This strategy treats answer engines as a discovery channel.

2. Training-limited access

This is the middle ground. Keep search and retrieval open, but block training-oriented agents where the policy or legal team does not want content used in future model development. The key is precision: blocking GPTBot or ClaudeBot is not the same as blocking OAI-SearchBot or Claude-SearchBot. A single blanket AI block can protect training rights and damage answer visibility at the same time.

3. Restrictive publisher access

This is a deliberate trade-off for publishers, data businesses or member-only sites whose content economics depend on licensing or direct traffic. It can be reasonable, but it should be treated as a content distribution decision, not as generic SEO hygiene. If the site opts out of major retrieval paths, stakeholders should expect lower eligibility for citations in those answer systems.

AI opt-out is not the same as crawler blocking

One common confusion is mixing product opt-outs with crawler rules. A Search Console control for Google AI experiences is not the same thing as blocking Googlebot, Google-Extended or other AI crawlers in robots.txt. A robots.txt rule is also not the same thing as a CDN rule that blocks traffic before the crawler can even read the file.

The clean way to decide is to ask which surface you are controlling. Are you controlling classic indexing, generative answer eligibility, model training, user-triggered page fetches, or server load? Each answer points to a different mechanism. This is why crawler policy belongs in an AEO operating document, not just in a developer ticket.

How to audit AI crawler access without false confidence

Read the live robots.txt file, not the CMS setting, and check every subdomain that hosts indexable content.
Search for wildcard disallow rules that catch AI bots not listed explicitly.
Compare robots.txt with CDN, WAF and bot-management settings; the edge can override the file.
Verify real crawler traffic through official IP endpoints or documented verification methods instead of trusting user-agent strings alone.
Check server logs for successful responses from search and retrieval bots on pages that should be citable.
Run a prompt portfolio before and after access changes so you can separate crawl eligibility from actual citation performance.
Track citation rate, mention rate and recommendation rate by engine; a crawler visit is not the same as being recommended.

The last two steps are the AEO part. Server logs can tell you whether a bot arrived. They cannot tell you whether the answer engine trusted the page, cited it, paraphrased it, ignored it or recommended a competitor. That requires repeated prompt sampling and source tracking by engine.

A simple robots.txt pattern for answer visibility

A visibility-first site can start from a simple pattern: allow the search and user-fetch agents that support answer retrieval, then make a separate policy decision for training agents. For example, a team may allow OAI-SearchBot, Claude-SearchBot and PerplexityBot on public guides, while deciding separately whether GPTBot, ClaudeBot and Google-Extended can access the same paths.

The important point is not the exact syntax in a blog post. The important point is that each user agent should be named intentionally. If the policy says 'we want to appear in ChatGPT search but not train future models', the robots.txt file and the WAF should express that difference instead of collapsing it into a single AI block.

What to monitor after changing crawler rules

Crawl access: which AI user agents reached important URLs and what status codes they received.
Index or retrieval symptoms: whether answer engines begin citing newly reachable URLs in relevant prompts.
Prompt-level visibility: citation rate, mention rate and recommendation rate by engine, not as one blended score.
Landing-page behavior: whether cited pages receive visible AI referrals, direct visits or branded-search lifts.
Load and abuse: whether any crawler creates latency, bandwidth pressure or repeated hits on low-value paths.
Policy drift: whether a hosting change, CDN rule or managed robots product silently changed the access decision.

Crawler access is eligibility. Citation tracking is evidence. Business impact is the reason to keep measuring after the technical fix.

For AEO, the mature position is not ideological. Some content should be open to answer retrieval. Some content may be excluded from training. Some paths should be blocked for security or commercial reasons. The winning setup is the one that makes those choices explicit, verifies them in logs, and measures whether they improve real visibility in the answer engines that matter to the business.

Frequently asked questions

Should I block AI crawlers?

Not as a blanket default if AI visibility matters. Block specific training agents only when that is a deliberate policy decision, and keep search or retrieval agents accessible when you want eligibility for AI citations.

Does allowing a crawler guarantee a citation?

No. Allowing a crawler only makes the page reachable. The page still has to be relevant, extractable, trustworthy and useful enough for the answer engine to cite or recommend.

Can I allow AI search but block AI training?

Often, yes. Many major AI systems expose different user agents or tokens for different purposes. The exact configuration depends on the platform, but the principle is to allow retrieval agents and make a separate decision on training-oriented agents.

Is llms.txt a replacement for robots.txt?

No. Robots.txt expresses crawl permissions to compliant bots. Llms.txt is a discovery and context file that can point AI systems toward useful site content. They solve different problems and should be maintained together.