Anthropic · User fetch

Claude-User

Fetches a page when someone asks Claude a question that needs it.

Operator
Anthropic
Type
User fetch · Fetch a page when a person asks an assistant about it. Because a human asked, operators say robots.txt may not apply.
robots.txt token
Claude-User (case-insensitive)
Follows robots.txt
Yes
User agent
Anthropic publishes the robots.txt token, not a full user agent string. Requests contain "Claude-User".

Verified against Anthropic’s documentation on November 21, 2026.

Should you block Claude-User?

Allow it. Claude-User only visits when a person asks about your page; blocking it means the assistant answers without your content.

If you block it: Claude can't retrieve your pages in response to user questions, which reduces your visibility in those answers.

robots.txt rules for Claude-User

A bot with its own group ignores your User-agent: * rules, so copy any Disallow lines you still need (like /cart) into its group.

Allow Claude-User
User-agent: Claude-User
Allow: /
Block Claude-User
User-agent: Claude-User
Disallow: /

How to verify a request is really Claude-User

Anyone can send a request that claims to be Claude-User. Anthropic publishes the IP ranges it uses: match the request's IP address against that list before trusting it, and before allowing it through a firewall.

Does your site let Claude-User in?

The free report checks your robots.txt against the major AI crawlers and tests whether your firewall lets them through.

Other Anthropic bots

Questions about Claude-User

What is Claude-User?

Claude-User is Anthropic's user fetch bot. Fetches a page when someone asks Claude a question that needs it.

Should I block Claude-User?

Allow it. Claude-User only visits when a person asks about your page; blocking it means the assistant answers without your content.

What happens if I block Claude-User?

Claude can't retrieve your pages in response to user questions, which reduces your visibility in those answers.

← All 20 AI crawlers