Common Crawl · Training

CCBot

Builds Common Crawl's open web archive, a free dataset that many AI models have been trained on.

Operator
Common Crawl
Type
Training · Collect content that may be used to train AI models. Blocking them doesn't remove you from AI search answers.
robots.txt token
CCBot (case-insensitive)
Follows robots.txt
Yes
User agent
CCBot/2.0 (https://commoncrawl.org/faq/)

Verified against Common Crawl’s documentation on November 21, 2026.

Should you block CCBot?

Your call. Blocking CCBot keeps your content out of future training data and doesn't remove you from AI search answers. Many stores allow it for brand familiarity; many publishers block it.

If you block it: New snapshots of your pages stop entering the Common Crawl dataset that model builders download.

robots.txt rules for CCBot

A bot with its own group ignores your User-agent: * rules, so copy any Disallow lines you still need (like /cart) into its group.

Allow CCBot
User-agent: CCBot
Allow: /
Block CCBot
User-agent: CCBot
Disallow: /

How to verify a request is really CCBot

Anyone can send a request that claims to be CCBot. Common Crawl publishes the IP ranges it uses: match the request's IP address against that list before trusting it, and before allowing it through a firewall.

Does your site let CCBot in?

The free report checks your robots.txt against the major AI crawlers and tests whether your firewall lets them through.

Questions about CCBot

What is CCBot?

CCBot is Common Crawl's training bot. Builds Common Crawl's open web archive, a free dataset that many AI models have been trained on.

Should I block CCBot?

Your call. Blocking CCBot keeps your content out of future training data and doesn't remove you from AI search answers. Many stores allow it for brand familiarity; many publishers block it.

What happens if I block CCBot?

New snapshots of your pages stop entering the Common Crawl dataset that model builders download.

← All 20 AI crawlers