AI crawler directory

Every AI crawler, what it does, and whether to block it

20 AI bots from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Meta and Common Crawl, with their user agents, IP lists and what blocking each one actually changes.

Checked against each operator’s own documentation on November 21, 2026.

Which of these can reach your site?

Four kinds of AI bot

The most common mistake is treating every AI bot the same. Blocking training crawlers is a reasonable choice; blocking search crawlers and fetchers is how sites vanish from AI answers.

Index pages so an assistant can find and cite them in answers. Blocking these is how sites disappear from AI search.

User-triggered fetchers

Fetch a page when a person asks an assistant about it. Because a human asked, operators say robots.txt may not apply.

Training crawlers

Collect content that may be used to train AI models. Blocking them doesn't remove you from AI search answers.

Control tokens

Not crawlers at all: robots.txt names that control how content collected by another crawler may be used.

Other AI agents

Specialised agents that don't fit the groups above.

A sensible robots.txt for online stores

Let assistants find and cite you; decide separately about training. Test it in the robots.txt tester before you publish.

robots.txt
# AI search and assistants: allow, so you can be found and cited.
# Copy any Disallow lines from your User-agent: * group in here too
# (like /cart or /checkout): a bot with its own group ignores the * rules.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# AI training: your choice. Delete this group to allow training.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

AI crawler questions

What's the difference between GPTBot and OAI-SearchBot?

GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot indexes pages so ChatGPT search can show and cite them. They're controlled separately, so you can block training (GPTBot) and still appear in ChatGPT answers (OAI-SearchBot).

Does blocking AI crawlers hurt my SEO?

Blocking training crawlers like GPTBot, ClaudeBot or Google-Extended doesn't affect Google or Bing rankings; Google states Google-Extended has no effect on Search. Blocking AI search crawlers like OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those assistants' answers, and blocking Googlebot or Bingbot removes you from search entirely.

How do I know a request really comes from an AI crawler?

User agent strings are easy to fake. Most operators publish the IP ranges their crawlers use; check the request's IP address against that list. Each crawler's page in this directory links to its official list.

Why do some AI fetchers ignore robots.txt?

User-triggered fetchers like ChatGPT-User and Perplexity-User visit a page because a person asked the assistant to. Their operators treat that like a browser visit, so they say robots.txt may not apply. To stop them, you'd need a firewall rule.