Document robots.txt AI agent access check

guides/geo.mdx

@@ -5,7 +5,7 @@
 keywords: ["Generative Engine Optimization", "GEO", "AI search", "Perplexity", "ChatGPT search", "Google AI Overviews", "AEO"]
 ---
 
 AI-powered tools like ChatGPT, Perplexity, and Google AI Overviews increasingly answer users' questions directly, citing sources rather than listing links. When a developer asks "how do I authenticate with [your product]," a well-optimized documentation page gets cited in the answer. A poorly structured page gets skipped—even if it ranks well in traditional search.
 
 Generative Engine Optimization (GEO) is the practice of structuring content so AI systems can understand, trust, and cite it accurately.
 
@@ -18,9 +18,9 @@
 | | SEO | GEO |
 |---|---|---|
 | Goal | Rank in search results | Get cited in AI-generated answers |
 | Key signals | Backlinks, keywords, page authority | Content accuracy, structure, directness |
 | User action | Click a link | Read an AI-generated answer |
 | Format preference | Any well-structured content | Scannable, question-answering content |
 
 The good news: the fundamentals overlap heavily. Accurate, well-structured content that directly answers questions performs well in both. GEO is less about tricks and more about writing clarity.
 
@@ -28,13 +28,13 @@
 
 AI answer engines evaluate content based on a few key factors:
 
 **Directness.** AI systems prioritize content that answers the question immediately. A page that buries the answer after three paragraphs of context is less likely to be cited than one that leads with the answer.
 
 **Accuracy and trust signals.** AI systems favor content from authoritative sources that appears factually reliable. For documentation, this means technical accuracy, consistent versioning, and content that matches what the product actually does.
 
 **Structural clarity.** Content that's logically organized—with meaningful headings, lists, and code blocks—is easier for AI systems to parse and excerpt correctly.
 
 **Specificity.** Vague, high-level content ("this feature is flexible and powerful") is less citable than specific, detailed content ("this endpoint returns a 429 status code when the rate limit of 100 requests per minute is exceeded").
 
 ## Write content AI systems can cite
 
@@ -55,7 +55,7 @@
 <!-- Buries the answer -->
 ## Authentication
 
 Authentication is an important part of using our API. Before you can make any requests, you'll need to understand how our authentication system works. Our API uses bearer tokens...
 ````
 
 ### Use headings that match questions
@@ -135,27 +135,57 @@
 
 ### LLMs.txt
 
 Mintlify automatically generates an `llms.txt` file for your documentation. LLMs.txt works similarly to `sitemap.xml` for traditional search—it provides AI systems with a structured index of your documentation. No configuration is required.
 
 You can view your LLMs.txt by appending `/llms.txt` to your documentation URL.
 
+### Allow AI agents in robots.txt
+
+Your `robots.txt` file controls which bots can crawl your site. If it blocks AI user agents, tools like ChatGPT, Claude, and Perplexity cannot read your documentation and will not cite it in answers.
+
+Mintlify's auto-generated `robots.txt` allows all crawlers by default. If you use a [custom robots.txt file](/optimize/seo#custom-sitemaps-and-robotstxt-files), make sure it does not block AI agents. The most common AI user agents include:
+
+- `GPTBot`, `OAI-SearchBot`, `ChatGPT-User` (OpenAI)
+- `ClaudeBot`, `Claude-User` (Anthropic)
+- `PerplexityBot` (Perplexity)
+- `Google-Extended` (Gemini)
+
+A robots.txt that blocks all crawlers also blocks AI agents:
+
+```txt Bad — blocks all AI agents
+User-agent: *
+Disallow: /
+```
+
+To block specific scrapers while allowing AI agents, target only the bots you want to restrict:
+
+```txt Good — allows AI agents
+User-agent: BadBot
+Disallow: /
+
+User-agent: *
+Disallow: /private/
+```
+
+Run [`mint score`](/cli/commands#mint-score) to check whether your site's `robots.txt` allows AI agents. The `robotsTxtAllowsAI` check passes when no AI user agents are blocked.
+
 ## Test how AI tools represent your docs
 
 Regularly test whether AI tools are citing your documentation accurately.
 
 Ask specific questions about your product in ChatGPT, Perplexity, and Claude:
 
 - "How do I authenticate API requests using [your product]?"
 - "What happens when I exceed the rate limit in [your product]?"
 - "Show me how to handle errors in [your product]'s API."
 
 Check the responses for:
 - Whether your documentation is cited at all
 - Whether the cited content is accurate
 - Whether the code examples are correct
 - Whether the AI is recommending the right approach
 
 When AI tools give wrong answers about your product, it often signals that your documentation is ambiguous, missing, or contradictory rather than that the AI is broken.
 
 ## Frequently asked questions
 
@@ -173,7 +203,7 @@
 </Accordion>
 
 <Accordion title="Should I write differently for AI versus human readers?">
 No. Content that's clear and direct for humans is clear and direct for AI systems. Optimizing specifically for AI at the expense of human readability is counterproductive and tends to produce content that's neither enjoyable to read nor well-cited. Write for your users first; GEO follows naturally from good technical writing.
 </Accordion>
 
 <Accordion title="What's llms.txt and do I need to configure it?">

optimize/seo.mdx

@@ -316,6 +316,10 @@
 
 To add a custom `sitemap.xml` or `robots.txt` file, create a `sitemap.xml` or `robots.txt` file at the root of your project. Adding a custom file overrides the automatically generated file of the same name. If you delete a custom file, the default file automatically applies again.
 
+<Warning>
+  If your custom `robots.txt` blocks AI user agents like `GPTBot`, `ClaudeBot`, or `PerplexityBot`, AI tools cannot crawl your documentation and will not cite it in answers. See [Allow AI agents in robots.txt](/guides/geo#allow-ai-agents-in-robotstxt) for details.
+</Warning>
+
 ## Disable indexing
 
 To prevent search engines from indexing a page, add `noindex: true` to the [frontmatter](/organize/pages) of the page.