There are two kinds of AI crawler, and confusing them is the most expensive mistake in this whole field.
- Retrieval crawlers build the live search indexes that AI engines cite from. Block one and you become uncitable in that engine, permanently, no matter how good your content is.
- Training crawlers collect data for training future models. Blocking these is a legitimate choice that has no effect on whether you get cited today.
They are independent switches. You can refuse training and remain fully citable — for most businesses that's exactly the right setting.
The crawlers, by job
| Company | Training | Live search index | User-triggered fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
Google-Extended | Googlebot | — | |
| Perplexity | — | PerplexityBot | — |
The user-triggered ones (ChatGPT-User, Claude-User) fetch a page because a person explicitly asked the assistant to look at that URL. Blocking them mostly just frustrates someone who was already trying to read your site.
A configuration that keeps you citable
This allows all retrieval, blocks all training. Adjust the training block to taste:
# ── Retrieval: allow these or you cannot be cited ──
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
# ── Training: optional, your call ──
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# ── Everything else ──
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
Never block Googlebot. That removes you from Google search and takes Gemini and AI Overviews with it.
The two traps that catch real businesses
The 2024 GPTBot advice. A wave of guidance told site owners to block GPTBot to stay out of AI training. Plenty did — often with a rule broad enough to catch more than intended. Blocking GPTBot alone is fine and doesn't affect citations. But a sweeping rule that also caught OAI-SearchBot silently removed those sites from ChatGPT.
The February 2026 change. Anthropic added Claude-SearchBot to its documentation on February 20, 2026. Any robots.txt written before that date predates the training/retrieval split for Claude entirely. If you blocked "Anthropic's crawler" in 2024, check what your rule actually matches now.
Should you block training?
A real tradeoff, not an obvious call.
Blocking training keeps your content out of future model weights. The cost is that future models are less likely to "know" your brand unprompted. For most local businesses this is a fine trade — retrieval does the work, and retrieval is unaffected.
Allowing training means a model may eventually recall your business without searching at all. For a brand playing a long game on name recognition, that has genuine upside.
There's no universally right answer. The wrong answer is doing it by accident.
How to check what you've actually got
- Open
yoursite.com/robots.txtin a browser. - Look for any
Disallowunder the retrieval crawlers above — and check whether aUser-agent: *block with a broadDisallowis catching them. - Check your server logs for
OAI-SearchBot,Claude-SearchBot, andPerplexityBot. If one has never visited, either it's blocked or it hasn't found you.
Also worth knowing: robots.txt is a request, not enforcement. Well-behaved crawlers from major AI companies respect it. If you need hard blocking, that's a server or CDN rule, not a text file.
Not sure what your robots.txt is doing? It's the first thing we check in any audit, and it takes minutes. Book a strategy call — the first four booked from the site are free.