A plain-English guide to Shopify robots.txt

Every Shopify store has a robots.txt file at yourstore.com/robots.txt. Most merchants never look at it, which is usually fine, because Shopify’s defaults are sensible. But it’s worth understanding what it does, because it’s now also where you decide which AI crawlers can read your store.

What robots.txt actually does

robots.txt is a set of instructions for well-behaved crawlers: which paths they may request and which they shouldn’t. Each group of rules starts with a User-agent line naming the crawler (or * for all of them), followed by Disallow and Allow lines.

Two things it does not do:

• It isn’t security. It’s a request, not a lock. Anything private needs real access control.
• It doesn’t remove pages from Google. A disallowed URL can still be indexed if other sites link to it, just without its content. To keep a page out of results, use a noindex tag and leave it crawlable so Google can see that tag.

What Shopify blocks by default

Shopify’s default file keeps crawlers away from pages that have no business in search results and would waste crawl budget. That includes checkout, cart and account pages, the admin, internal search results, and the near-duplicate URLs created by filtering and sorting collections. A trimmed example:

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search
Disallow: /collections/*sort_by*
Disallow: /collections/*+*
Sitemap: https://yourstore.com/sitemap.xml

Note the last line. The file also points crawlers at your sitemap, which is how they discover your products, collections, pages and articles.

When it’s worth changing

For most stores, the honest answer is rarely. Good reasons include:

• An app or theme feature creating large numbers of thin or duplicate URLs that crawlers keep hitting.
• Internal pages, like a wholesale price list, that shouldn’t be crawled.
• A deliberate policy on AI crawlers (more below).

Shopify lets you customize the file with a robots.txt.liquid theme template, which edits theme code. An app with a robots.txt editor lets you add and remove rules without touching the theme.

AI crawlers

AI companies now run their own crawlers, and most document a user agent you can address in robots.txt. They generally fall into two groups:

• Search and answer crawlers fetch pages to answer a user’s question right now, often with a link back. Examples include OAI-SearchBot and PerplexityBot.
• Training crawlers collect content to train future models. Examples include GPTBot, ClaudeBot and CCBot. Google uses a separate token, Google-Extended, to let you opt out of Gemini training without affecting Google Search.

If you want your store to appear in AI answers, don’t block the search crawlers. Whether to allow training crawlers is a separate business decision. Many stores allow both, and some block training only:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User agent names and their purposes change, so check each company’s current documentation before relying on a rule. There’s more on how assistants pick stores in How AI assistants decide which stores to mention.

How to check yours

1. Open yourstore.com/robots.txt in a browser and read it top to bottom.
2. Look for any rule that disallows /products, /collections or /.
3. Check which AI crawlers are named, and whether that matches what you intend.
4. In Google Search Console, look for pages reported as “Blocked by robots.txt” that you expected to rank.