Google · AI search

Googlebot

Crawls for Google Search, including all Search features (such as AI Overviews), Discover, Images and News.

Operator
Google
Type
AI search · Index pages so an assistant can find and cite them in answers. Blocking these is how sites disappear from AI search.
robots.txt token
Googlebot (case-insensitive)
Follows robots.txt
Yes
User agent
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Verified against Google’s documentation on November 21, 2026.

Should you block Googlebot?

Allow it. Googlebot is how Google finds pages to show and cite. For an online store there's rarely a good reason to block it.

If you block it: Your pages drop out of Google Search, and with it Google's AI features built on Search.

robots.txt rules for Googlebot

A bot with its own group ignores your User-agent: * rules, so copy any Disallow lines you still need (like /cart) into its group.

Allow Googlebot
User-agent: Googlebot
Allow: /
Block Googlebot
User-agent: Googlebot
Disallow: /

How to verify a request is really Googlebot

Anyone can send a request that claims to be Googlebot. Google publishes the IP ranges it uses: match the request's IP address against that list before trusting it, and before allowing it through a firewall.

Does your site let Googlebot in?

The free report checks your robots.txt against the major AI crawlers and tests whether your firewall lets them through.

Other Google bots

Questions about Googlebot

What is Googlebot?

Googlebot is Google's ai search bot. Crawls for Google Search, including all Search features (such as AI Overviews), Discover, Images and News.

Should I block Googlebot?

Allow it. Googlebot is how Google finds pages to show and cite. For an online store there's rarely a good reason to block it.

What happens if I block Googlebot?

Your pages drop out of Google Search, and with it Google's AI features built on Search.

← All 20 AI crawlers