GPTBot, ClaudeBot, PerplexityBot: which AI crawlers to allow, and what each one does

Blocking the retrieval crawlers is how you become invisible in AI answers while keeping your training-data objection, and most sites block the wrong ones. Each vendor runs distinct bots for training, for its search index, and for live user-triggered fetches. You can refuse training and still be citable, but only if you allow the retrieval bots by name.

The short version: OpenAI runs three distinct bots. GPTBot collects training data, OAI-SearchBot builds the ChatGPT search index, and ChatGPT-User performs live fetches when a conversation triggers a web search. Blocking GPTBot does not affect whether ChatGPT cites you (OpenAI bot documentation, as of August 2026). The same split, under different names, exists at Anthropic, Perplexity, Google, Apple, and Meta. Every claim below was checked against the operator’s published documentation in August 2026; anything a vendor does not document is marked not documented.

Which AI crawler does what?

“Honors robots.txt” below means the vendor’s published documentation says it does. Live-fetch agents are a special case across the board: OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User “generally ignores” robots.txt, because a human asked for the page. All rows as of August 2026.

Bot	Operator	Role	Honors robots.txt	Verification
GPTBot	OpenAI	Training data collection	Yes, per OpenAI docs	IP list (gptbot.json)
OAI-SearchBot	OpenAI	ChatGPT search index	Yes, per OpenAI docs	IP list (searchbot.json)
ChatGPT-User	OpenAI	Live fetch on a user’s request	Not guaranteed; “robots.txt rules may not apply”	IP list (chatgpt-user.json)
ClaudeBot	Anthropic	Training data collection	Yes, per Anthropic docs	IP list (claude.com/crawling/bots.json)
Claude-SearchBot	Anthropic	Search result quality	Yes, per Anthropic docs	Same IP list
Claude-User	Anthropic	Live fetch during conversations	Yes, per Anthropic docs	Same IP list
PerplexityBot	Perplexity	Search surfacing; explicitly not model training	Yes, per Perplexity docs	IP list (perplexitybot.json)
Perplexity-User	Perplexity	Live fetch on a user’s query	No; “generally ignores robots.txt”	IP list (perplexity-user.json)
Googlebot	Google	Search index, which also serves AI Overviews and AI Mode	Yes	Published IP ranges + reverse DNS
Google-Extended	Google	Not a crawler: a robots.txt token controlling Gemini/Vertex training and grounding	Control token only	n/a
Applebot	Apple	Siri, Spotlight and Apple search; renders JavaScript	Yes, per Apple docs	Published verification
Applebot-Extended	Apple	Not a crawler: a token controlling whether crawled content trains Apple’s models	Control token only	n/a
bingbot	Microsoft	Bing index, which grounds Copilot answers	Yes	Reverse DNS (*.search.msn.com)
CCBot	Common Crawl	Open crawl dataset used by many training pipelines	Yes, per Common Crawl docs	IP list (index.commoncrawl.org/ccbot.json)
meta-externalagent	Meta	AI training and direct content indexing	Yes, per Meta docs	Documented UA strings
meta-externalfetcher	Meta	Live fetch at a user’s request	No; “may bypass robots.txt”	Documented UA strings
Bytespider	ByteDance	Training data collection	Not verifiable; see below	No official docs or IP list

Two rows deserve a closer look, because they are not crawlers at all. Google-Extended has no user-agent of its own: crawling happens with Google’s existing agents, and the token only controls whether your content is used to train and ground Gemini apps and Vertex AI. Applebot-Extended works the same way for Apple: it never fetches a page; it tells Apple not to use content Applebot already crawled for training its foundation models, and disallowing it does not remove you from Apple’s search results.

Bytespider is the opposite problem. ByteDance publishes no official documentation, no IP list, and no robots.txt statement, and independent server-log analyses through the first half of 2026 report it fetching pages that robots.txt disallows. Add a disallow line if you want, but treat compliance as unenforceable.

If I block GPTBot, does ChatGPT stop citing me?

No. GPTBot collects training data. OAI-SearchBot and ChatGPT-User are the bots that determine whether your content appears in ChatGPT answers when web search is active. Blocking GPTBot leaves both untouched.

There is a second-order effect worth knowing: only about 18% of ChatGPT conversations trigger a web search at all; the rest are answered from model weights. So for ChatGPT specifically, your presence in training data matters more than most robots.txt guides admit, and the retrieval bots matter comparatively more for Perplexity, which grounds nearly every answer in a live web result. A full training opt-out is a bigger decision than it looks: you are trading away the 82% of answers that come from what the model already knows.

The practical rule: to opt out of training while staying citable, block GPTBot and leave OAI-SearchBot and ChatGPT-User alone.

Does blocking Google-Extended remove me from AI Overviews?

No. Google is explicit that AI is built into Search, and that “robots.txt directives for Googlebot” are the control for how sites are crawled for Search, AI Overviews included. Google-Extended controls a different thing: whether your content trains and grounds Gemini apps and the Vertex AI API. Google also states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”

This is the most common misconfiguration we see. A site adds Disallow: / under User-agent: Google-Extended, assumes it has opted out of Google’s AI features, and is surprised to still appear in AI Overviews, or, more often, to vanish from Gemini answers it wanted. If you want less AI Overviews exposure, the levers are the ordinary Search ones: nosnippet, data-nosnippet, max-snippet, or noindex. There is no robots.txt token that targets AI Overviews directly, as of August 2026.

Which bots can actually read my page?

Most AI crawlers execute zero JavaScript. GPTBot, ClaudeBot, PerplexityBot, Bytespider and Meta’s crawlers fetch raw HTML only. The documented exceptions are Googlebot and Applebot, which render pages. If your marketing site or docs are client-side rendered, the no-JS bots see a near-empty page regardless of what your robots.txt says.

That makes rendering the prerequisite. Allowing a bot that cannot read your page is allowing nothing: if your site is a single-page app whose content loads via JavaScript, the first fix is server-side rendering or static generation, not robots.txt. See what AI crawlers read on your site for the rendering breakdown, and schema markup for AI visibility for what to add once bots can see the page.

What should a B2B SaaS that wants to be cited put in robots.txt?

The goal for most B2B SaaS sites: refuse training, allow retrieval and indexing. A minimal block that does that, as of August 2026:

# Opt out of AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

# Retrieval and citation bots stay allowed by default:
# OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
# PerplexityBot, Perplexity-User, Googlebot, Applebot, bingbot

Three notes on this block. You do not need Allow: / lines for bots you want; allow is the default, and explicit allows only matter under a broader disallow. Blocking meta-externalagent opts you out of Meta’s AI training but also its direct content indexing, per Meta’s documentation; that is the trade. And the Bytespider line is a signal, not a control, for the reasons above.

How do I check which bots are actually visiting?

Grep your recent access logs for the user-agent names. On a standard nginx or Apache setup:

for bot in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User \
  PerplexityBot Googlebot bingbot CCBot Bytespider; do
  printf "%s: " "$bot"; grep -c "$bot" /var/log/nginx/access.log
done

What to look for: the retrieval bots you want (OAI-SearchBot, PerplexityBot, bingbot) should appear and should not be getting 403s; the training bots you blocked should be absent or refused; any bot getting a 200 that you meant to block is a misconfiguration. To confirm a bot is genuine rather than spoofed, use the vendor’s published verification: OpenAI, Anthropic, Perplexity and Common Crawl publish IP lists (linked in the table above), and Google and Microsoft document reverse-DNS checks. Check the vendor’s page directly; IP ranges change faster than any article about them.

One thing to do today

Open your robots.txt and run the log check above for the last seven days. Confirm the retrieval bots you want are neither disallowed nor returning 403s. It takes under thirty minutes and it is the highest-leverage robots.txt check there is. How the engines choose among the pages they can reach is its own topic; see how five AI engines decide what to cite.

Then run a free scan to see which engines currently find and cite you; it takes about 90 seconds. If one engine is absent, robots.txt is the first place to look.