CCBot
Builds Common Crawl's open web archive, a free dataset that many AI models have been trained on.
- Operator
- Common Crawl
- Type
- Training · Collect content that may be used to train AI models. Blocking them doesn't remove you from AI search answers.
- robots.txt token
CCBot(case-insensitive)- Follows robots.txt
- Yes
- User agent
CCBot/2.0 (https://commoncrawl.org/faq/)- IP ranges
- index.commoncrawl.org/ccbot.json
- Official docs
- commoncrawl.org/ccbot
Verified against Common Crawl’s documentation on November 21, 2026.
Should you block CCBot?
Your call. Blocking CCBot keeps your content out of future training data and doesn't remove you from AI search answers. Many stores allow it for brand familiarity; many publishers block it.
If you block it: New snapshots of your pages stop entering the Common Crawl dataset that model builders download.
robots.txt rules for CCBot
A bot with its own group ignores your User-agent: * rules, so copy any Disallow lines you still need (like /cart) into its group.
User-agent: CCBot
Allow: /User-agent: CCBot
Disallow: /How to verify a request is really CCBot
Anyone can send a request that claims to be CCBot. Common Crawl publishes the IP ranges it uses: match the request's IP address against that list before trusting it, and before allowing it through a firewall.
Does your site let CCBot in?
The free report checks your robots.txt against the major AI crawlers and tests whether your firewall lets them through.
Questions about CCBot
What is CCBot?
CCBot is Common Crawl's training bot. Builds Common Crawl's open web archive, a free dataset that many AI models have been trained on.
Should I block CCBot?
Your call. Blocking CCBot keeps your content out of future training data and doesn't remove you from AI search answers. Many stores allow it for brand familiarity; many publishers block it.
What happens if I block CCBot?
New snapshots of your pages stop entering the Common Crawl dataset that model builders download.