Why robots.txt matters for AI visibility
Your robots.txt file controls which crawlers can access your website. Most site owners are familiar with configuring it for Googlebot and Bingbot, but a new class of AI crawlers now regularly indexes the web to train models and power real-time AI search.
If your robots.txt blocks these AI crawlers — or fails to explicitly permit them — your site may be invisible to AI-powered search systems, regardless of how good your content is.
This guide covers the major AI crawlers you need to know, how to configure your robots.txt, and templates you can use immediately.
Major AI crawlers and their user agents
The following AI crawlers are actively indexing the web as of 2025:
| Crawler | User Agent | Platform | Purpose |
|---|---|---|---|
| GPTBot | GPTBot | OpenAI / ChatGPT | Training data & real-time search |
| ClaudeBot | ClaudeBot | Anthropic / Claude | Training data |
| PerplexityBot | PerplexityBot | Perplexity AI | Real-time search |
| Google-Extended | Google-Extended | Google Gemini/AI Overview | AI training opt-out signal |
| OAI-SearchBot | OAI-SearchBot | OpenAI search | Real-time web search |
| Amazonbot | Amazonbot | Amazon Alexa AI | Training & search |
| Bytespider | Bytespider | ByteDance / Doubao | Training data |
| Applebot | Applebot | Apple / Siri | Training & search |
Note: Google-Extended is the user agent specifically for Google's generative AI products (Gemini, AI Overviews). Blocking it does not affect your Google Search rankings — only your inclusion in AI-generated content.
Template: Allow all AI crawlers (recommended)
If you want maximum AI visibility — the choice for most marketing and content sites — use this template:
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/# AI crawlers — explicitly permitted User-agent: GPTBot Allow: /
User-agent: OAI-SearchBot Allow: /
User-agent: ClaudeBot Allow: /
User-agent: PerplexityBot Allow: /
User-agent: Google-Extended Allow: /
User-agent: Amazonbot Allow: /
User-agent: Bytespider Allow: /
User-agent: Applebot Allow: /
Sitemap: https://yourdomain.com/sitemap.xml ```
Explicitly listing AI crawlers even when your wildcard rule allows them is a best practice — it makes your intent clear and ensures future robots.txt parsers interpret your directives correctly.
Template: Allow search AI, block training crawlers
If you want your content to appear in AI search results but opt out of being used as AI training data, use a selective configuration:
User-agent: *
Allow: /
Disallow: /api/# Block training-only crawlers User-agent: GPTBot Disallow: /
User-agent: ClaudeBot Disallow: /
User-agent: Bytespider Disallow: /
# Allow real-time search AI User-agent: PerplexityBot Allow: /
User-agent: OAI-SearchBot Allow: /
User-agent: Google-Extended Allow: /
Sitemap: https://yourdomain.com/sitemap.xml ```
Important caveat: The distinction between "training" and "search" crawlers is not always clean. OpenAI's GPTBot is used for both training and powering ChatGPT's Browse feature. Blocking it may reduce your visibility in ChatGPT answers even if OAI-SearchBot is allowed.
Common mistakes to avoid
Mistake 1: Wildcard disallow without AI exceptions.
If you have Disallow: / for any reason — such as a staging environment — make sure AI crawlers are explicitly permitted if you want AI visibility.
Mistake 2: Using outdated user agent names. AI crawlers update their user agents. Always check the official documentation for each crawler to ensure you're using current names.
Mistake 3: No Sitemap directive.
Always include a Sitemap: directive pointing to your sitemap.xml. This helps all crawlers — AI and traditional — discover your content efficiently.
Mistake 4: Blocking /api/ but serving crawlable content there. If your site renders content via API routes (common in Next.js applications), ensure you're blocking only private API endpoints, not content routes.
Mistake 5: Never testing your robots.txt. Use Google Search Console's robots.txt tester or direct URL access to verify your file is serving correctly before assuming it's working.
Verifying your configuration
After updating your robots.txt:
1. Access it directly: Visit https://yourdomain.com/robots.txt and verify the content is correct.
2. Check content-type: It should be served as text/plain.
3. Test in Google Search Console: Use the URL Inspection tool to verify Googlebot access.
4. Run an Auditly audit: Auditly specifically checks for AI crawler permissions in your robots.txt as part of the AI Readiness score and will flag any crawlers that are blocked or missing explicit permissions.