robots.txt tester for AI crawlers
Paste your robots.txt, pick a crawler and a path, and see exactly which rule applies. Same matching rules Google and OpenAI follow; nothing leaves your browser.
/wholesale: it’s allowed, because a crawler with its own group ignores every rule under User-agent: *. Edit the file to see how rules interact.How crawlers read robots.txt
- Pick a group. The crawler looks for a group naming it (case-insensitive). Only if there isn’t one does it use
User-agent: *. - Find matching rules. Each Allow or Disallow path is a prefix;
*matches anything and$anchors the end. - Longest match wins. The most specific rule decides. On a tie, Allow beats Disallow.
- No match means allowed. An empty
Disallow:blocks nothing.
The mistakes that hide stores from AI search
- Blocking OAI-SearchBot or PerplexityBot while meaning to block training crawlers like GPTBot.
- Adding a named group for a bot and forgetting it no longer inherits your
*rules. - A blanket
Disallow: /left over from a staging site.
Not sure what a bot is for? Look it up in the AI crawler directory, or read the Shopify robots.txt guide.
Check your whole site in one go
The free AI visibility report tests robots.txt, your firewall, llms.txt, schema and product pages, and writes the fixes.
Questions
Why is a bot allowed on a path my * group disallows?
A crawler obeys only the most specific group that names it. If robots.txt has a group for OAI-SearchBot, that bot ignores every rule under User-agent: *. Copy any Disallow lines you still need into its own group.
Which rule wins when Allow and Disallow both match?
The longest matching rule wins, because it's the most specific. If an Allow and a Disallow rule are the same length, Allow wins. This is how Google, Bing and the RFC 9309 standard handle it.
Does robots.txt stop AI crawlers?
Reputable crawlers like GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot follow it. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not, because a person asked for the page. robots.txt is a request, not access control.
More free AI tools
- AI visibility reportCrawler access, firewall test, AI files, homepage and product pages, with copy-paste fixes.
- AI crawler directoryWhat every AI bot does, its user agent and IP list, and whether to block it.
- llms.txt generatorBuild a valid llms.txt that summarises your site for AI assistants.
- AI readiness checklistFind where the chain from crawl to recommendation breaks for your store.