Claude-SearchBot
Navigates the web to improve the quality of Claude's search results.
- Operator
- Anthropic
- Type
- AI search · Index pages so an assistant can find and cite them in answers. Blocking these is how sites disappear from AI search.
- robots.txt token
Claude-SearchBot(case-insensitive)- Follows robots.txt
- Yes
- User agent
- Anthropic publishes the robots.txt token, not a full user agent string. Requests contain "Claude-SearchBot".
- IP ranges
- claude.com/crawling/bots.json
- Official docs
- support.claude.com/en/articles/8896518
Verified against Anthropic’s documentation on November 21, 2026.
Should you block Claude-SearchBot?
Allow it. Claude-SearchBot is how Anthropic finds pages to show and cite. For an online store there's rarely a good reason to block it.
If you block it: Your content isn't indexed for Claude's search, which can reduce your visibility and accuracy in Claude's answers.
robots.txt rules for Claude-SearchBot
A bot with its own group ignores your User-agent: * rules, so copy any Disallow lines you still need (like /cart) into its group.
User-agent: Claude-SearchBot
Allow: /User-agent: Claude-SearchBot
Disallow: /How to verify a request is really Claude-SearchBot
Anyone can send a request that claims to be Claude-SearchBot. Anthropic publishes the IP ranges it uses: match the request's IP address against that list before trusting it, and before allowing it through a firewall.
Does your site let Claude-SearchBot in?
The free report checks your robots.txt against the major AI crawlers and tests whether your firewall lets them through.
Other Anthropic bots
Questions about Claude-SearchBot
What is Claude-SearchBot?
Claude-SearchBot is Anthropic's ai search bot. Navigates the web to improve the quality of Claude's search results.
Should I block Claude-SearchBot?
Allow it. Claude-SearchBot is how Anthropic finds pages to show and cite. For an online store there's rarely a good reason to block it.
What happens if I block Claude-SearchBot?
Your content isn't indexed for Claude's search, which can reduce your visibility and accuracy in Claude's answers.