Anthropic · AI search

Claude-SearchBot

Navigates the web to improve the quality of Claude's search results.

Operator
Anthropic
Type
AI search · Index pages so an assistant can find and cite them in answers. Blocking these is how sites disappear from AI search.
robots.txt token
Claude-SearchBot (case-insensitive)
Follows robots.txt
Yes
User agent
Anthropic publishes the robots.txt token, not a full user agent string. Requests contain "Claude-SearchBot".

Verified against Anthropic’s documentation on November 21, 2026.

Should you block Claude-SearchBot?

Allow it. Claude-SearchBot is how Anthropic finds pages to show and cite. For an online store there's rarely a good reason to block it.

If you block it: Your content isn't indexed for Claude's search, which can reduce your visibility and accuracy in Claude's answers.

robots.txt rules for Claude-SearchBot

A bot with its own group ignores your User-agent: * rules, so copy any Disallow lines you still need (like /cart) into its group.

Allow Claude-SearchBot
User-agent: Claude-SearchBot
Allow: /
Block Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: /

How to verify a request is really Claude-SearchBot

Anyone can send a request that claims to be Claude-SearchBot. Anthropic publishes the IP ranges it uses: match the request's IP address against that list before trusting it, and before allowing it through a firewall.

Does your site let Claude-SearchBot in?

The free report checks your robots.txt against the major AI crawlers and tests whether your firewall lets them through.

Other Anthropic bots

Questions about Claude-SearchBot

What is Claude-SearchBot?

Claude-SearchBot is Anthropic's ai search bot. Navigates the web to improve the quality of Claude's search results.

Should I block Claude-SearchBot?

Allow it. Claude-SearchBot is how Anthropic finds pages to show and cite. For an online store there's rarely a good reason to block it.

What happens if I block Claude-SearchBot?

Your content isn't indexed for Claude's search, which can reduce your visibility and accuracy in Claude's answers.

← All 20 AI crawlers