robots.txt
A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.
GEO AI
llms.txt
A proposed Markdown file at a site's root listing its most important content for language models; a community convention, not a standard, and not used by Google Search.
In full
Proposed in September 2024 with a specification at llmstxt.org, llms.txt is a curated index in Markdown intended to help models find authoritative pages without parsing navigation and boilerplate. It has seen meaningful adoption among documentation and developer-tool sites, but no major model provider has committed to consistently fetching it, and Google's official generative AI guidance states Google Search does not use llms.txt and it neither helps nor harms rankings. It is cheap to publish and reasonable for documentation sites; it is not an AI visibility strategy.
Example
A developer platform publishes `/llms.txt` linking its API reference, quickstarts and changelog with one-line descriptions each.
Related terms
Generative Engine Optimization
The practice of improving how often and how accurately a brand's content is retrieved, cited and represented in AI-generated answers.
AI crawler
An automated agent operated by an AI company to fetch web content for model training, search indexing, or live retrieval when answering a…
Model training opt-out
Mechanisms by which a publisher signals that its content should not be used to train AI models, principally robots.txt tokens and…
GPTBot
OpenAI's crawler used to collect publicly available web content that may be used to improve future models; controllable via robots.txt.
Where to read more
- The /llms.txt File · Standard · llmstxt.org