Every AI crawler, what it does, and whether to block it
20 AI bots from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Meta and Common Crawl, with their user agents, IP lists and what blocking each one actually changes.
Checked against each operator’s own documentation on November 21, 2026.
Which of these can reach your site?
Four kinds of AI bot
The most common mistake is treating every AI bot the same. Blocking training crawlers is a reasonable choice; blocking search crawlers and fetchers is how sites vanish from AI answers.
AI search crawlers
Index pages so an assistant can find and cite them in answers. Blocking these is how sites disappear from AI search.
- OAI-SearchBotOpenAI · Follows robots.txtSurfaces websites in search results in ChatGPT's search features.
- Claude-SearchBotAnthropic · Follows robots.txtNavigates the web to improve the quality of Claude's search results.
- PerplexityBotPerplexity · Follows robots.txtSurfaces and links websites in Perplexity search results. Perplexity says it isn't used to crawl content for training foundation models.
- GooglebotGoogle · Follows robots.txtCrawls for Google Search, including all Search features (such as AI Overviews), Discover, Images and News.
- BingbotMicrosoft · Follows robots.txtCrawls for Bing search, which also grounds Microsoft Copilot's web answers.
- ApplebotApple · Follows robots.txtCrawls for search features in Spotlight, Siri and Safari.
- Amzn-SearchBotAmazon · Follows robots.txtImproves search experiences in Amazon products and services. Amazon says it doesn't train generative AI.
User-triggered fetchers
Fetch a page when a person asks an assistant about it. Because a human asked, operators say robots.txt may not apply.
- ChatGPT-UserOpenAI · May ignore robots.txtVisits a page when a ChatGPT user or a custom GPT asks for it.
- Claude-UserAnthropic · Follows robots.txtFetches a page when someone asks Claude a question that needs it.
- Perplexity-UserPerplexity · May ignore robots.txtVisits a page when a Perplexity user's question needs it. Not used for crawling or training.
- Amzn-UserAmazon · May ignore robots.txtSupports user actions, such as answering Alexa questions. Amazon says it doesn't train generative AI.
- meta-externalfetcherMeta · May ignore robots.txtFetches individual links at a user's request, including for Meta's AI agent features.
Training crawlers
Collect content that may be used to train AI models. Blocking them doesn't remove you from AI search answers.
- GPTBotOpenAI · Follows robots.txtCrawls content that may be used to train OpenAI's generative AI foundation models.
- ClaudeBotAnthropic · Follows robots.txtCollects public web content that may be used to train and improve Anthropic's models.
- AmazonbotAmazon · Follows robots.txtImproves Amazon's products and services, and its data may be used to train Amazon's AI models.
- meta-externalagentMeta · Follows robots.txtCrawls for uses such as training foundation AI models or improving products by indexing content directly.
- CCBotCommon Crawl · Follows robots.txtBuilds Common Crawl's open web archive, a free dataset that many AI models have been trained on.
Control tokens
Not crawlers at all: robots.txt names that control how content collected by another crawler may be used.
Other AI agents
Specialised agents that don't fit the groups above.
A sensible robots.txt for online stores
Let assistants find and cite you; decide separately about training. Test it in the robots.txt tester before you publish.
# AI search and assistants: allow, so you can be found and cited.
# Copy any Disallow lines from your User-agent: * group in here too
# (like /cart or /checkout): a bot with its own group ignores the * rules.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# AI training: your choice. Delete this group to allow training.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /AI crawler questions
What's the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot indexes pages so ChatGPT search can show and cite them. They're controlled separately, so you can block training (GPTBot) and still appear in ChatGPT answers (OAI-SearchBot).
Does blocking AI crawlers hurt my SEO?
Blocking training crawlers like GPTBot, ClaudeBot or Google-Extended doesn't affect Google or Bing rankings; Google states Google-Extended has no effect on Search. Blocking AI search crawlers like OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those assistants' answers, and blocking Googlebot or Bingbot removes you from search entirely.
How do I know a request really comes from an AI crawler?
User agent strings are easy to fake. Most operators publish the IP ranges their crawlers use; check the request's IP address against that list. Each crawler's page in this directory links to its official list.
Why do some AI fetchers ignore robots.txt?
User-triggered fetchers like ChatGPT-User and Perplexity-User visit a page because a person asked the assistant to. Their operators treat that like a browser visit, so they say robots.txt may not apply. To stop them, you'd need a firewall rule.