Definition
The `robots.txt` file (Robot Exclusion Standard) is a plain-text protocol placed at the root of a domain (`yourdomain.com/robots.txt`) that communicates crawling permissions to all bots that respect the standard. It uses `User-agent` directives to target specific bots and `Allow` / `Disallow` rules to control access.
In the AI era, `robots.txt` has new importance: AI training crawlers (GPTBot, Google-Extended, CCBot, ClaudeBot, Applebot-Extended) all check `robots.txt` before fetching content. Misconfigured `robots.txt` files can accidentally block AI systems from reading your pages, eliminating citation opportunities.
Why It Matters
Accidental `Disallow: /` entries or overly aggressive bot-blocking are among the most common causes of poor AI readiness scores. Your robots.txt directly determines whether AI systems can access and cite your content.
Examples
- Allowing GPTBot: `User-agent: GPTBot` / `Allow: /`
- Blocking training on sensitive pages: `Disallow: /internal/`
- Linking to your sitemap: `Sitemap: https://yourdomain.com/sitemap.xml`
- Checking AI crawler access with the robots.txt tester