More from Sourceable
Continue reading our latest insights
Continue reading our latest insights
llms.txt is a proposed Markdown file that website owners can place at the root of their domain, such as:
https://example.com/llms.txtThe idea is simple. The file gives AI systems a short, structured overview of a website. It can point to important pages, documentation, and other useful resources.
The llms.txt proposal helps language models and AI agents understand websites more easily. The specification recommends a Markdown structure with a site name, a short description, and links to important resources.
But there is an important distinction:
llms.txt is not the same as robots.txt.
robots.txt controls crawler access. llms.txt provides information and context about a website.
The two files solve different problems.
robots.txt tells crawlers which areas they can or cannot access.
llms.txt is a proposed way to give AI systems a concise map of important website content.
For example, robots.txt may say:
Do not crawl this directory.
An llms.txt file may instead point an AI system toward:
Product documentation
API documentation
Important guides
Research
Official resources
Key pages about the company
OpenAI also provides separate crawler controls through robots.txt. For example, website owners can manage access for OAI-SearchBot and GPTBot independently.
So publishing llms.txt does not replace your robots.txt file.
This is where many articles make an important mistake.
Google currently says that Google Search and its generative AI features do not require llms.txt. Google also says that these files do not provide a special ranking advantage.
Google recommends focusing on standard SEO fundamentals instead. Make sure search engines can crawl your pages and find important content. Provide useful content and keep key information in text form.
That means you should not publish an llms.txt file expecting it to automatically increase your Google rankings.
Think of it as an additional machine-readable resource, not an SEO shortcut.
The web is changing.
AI assistants and agents increasingly interact with websites to find information. The llms.txt proposal addresses this change.
A website can contain hundreds or thousands of pages. Some pages may contain outdated, duplicate, or promotional information. Others may be less useful than the main documentation.
An llms.txt file can provide a concise list of pages that a site owner considers important.
For example, a SaaS company could use it to highlight:
Product documentation
API references
Integration guides
Security information
Product overview
Technical documentation
The goal is to make important information easier for machines to discover and interpret.
However, adoption and support vary across AI systems. The llms.txt specification itself remains a proposal rather than a universal web standard.
The current llms.txt proposal uses Markdown.
A basic file can include:
1. Website or project name
Start with an H1 containing the name of your website or project.
2. Short description
Add a clear description that explains what the website does.
3. Important information
Include useful context that helps an AI system understand the site.
4. Important URLs
Add links to key documentation, guides, resources, or other pages.
A simple example could look like this:
# Example SaaS
> Example SaaS helps businesses manage customer support.
## Documentation
- [Getting Started](https://example.com/docs/getting-started)
- [API Reference](https://example.com/docs/api)
- [Integrations](https://example.com/integrations)
## Resources
- [Blog](https://example.com/blog)
- [Security](https://example.com/security)
The official proposal defines the structure more specifically, including the required H1 and optional sections with Markdown links.
Creating the file does not need to be complicated. You can use an llms.txt generator to create the basic file structure faster.
First, identify the pages that best explain your website. Focus on original documentation, product information, research, and useful resources.
Next, organize those URLs into clear sections.
Then create a Markdown file named:
llms.txt
Finally, publish it at:
https://yourdomain.com/llms.txtBefore publishing, check every URL and make sure the information is accurate.
If your website changes often, review the file regularly. Remove old pages that are no longer key resources.
No.
Publishing llms.txt does not guarantee that ChatGPT, Gemini, Claude, Perplexity, or another AI system will mention or cite your brand.
It also does not guarantee that an AI system will use the URLs inside the file.
AI systems use different crawling, retrieval, ranking, and response systems. Their behavior can change over time.
For AI visibility, your broader website matters much more than one file.
Your content should be useful, clear, trustworthy, well structured, and easy for relevant systems to access.
llms.txt should be viewed as one small part of a wider AI visibility strategy.
A strong AI-ready website should have:
Useful and original content
Clear site structure
Strong internal links
Accessible text
Reliable technical documentation
Appropriate crawler permissions
Accurate structured data where relevant
Clear information about products and services
Regular monitoring of how AI systems represent the brand
This is where AI visibility becomes important.
Instead of only asking whether your website has an llms.txt file, businesses should also ask:
Does AI actually understand and mention my brand?
That is a measurable question.
For most websites, llms.txt is optional.
If you have useful documentation or a large website, an llms.txt file can help organize important resources. AI systems can choose whether to use the file.
But it should not replace SEO fundamentals.
Google specifically recommends continuing to use established SEO practices for visibility in both traditional Search and generative AI experiences.
So the better approach is:
Build a strong website first. Use llms.txt as an additional resource. Monitor what AI systems actually do with your content.
What is llms.txt?
llms.txt is a proposed Markdown file that gives AI systems a structured overview of a website and links to important resources.
Where should llms.txt be placed?
The standard proposal places the file at the root of a website:
https://yourdomain.com/llms.txtIs llms.txt the same as robots.txt?
No. robots.txt controls crawler access. llms.txt provides information and context about important website content.
Does Google use llms.txt?
Google's current documentation says Google Search does not use llms.txt as a special optimization method. Google also does not require llms.txt for visibility in Search or its generative AI features.
Does llms.txt improve AI visibility?
It may provide useful context to AI systems that support the format. However, publishing the file does not guarantee more AI mentions or citations.
Do I need llms.txt for SEO?
No. Google recommends focusing on standard SEO fundamentals rather than relying on llms.txt as a ranking tactic.
llms.txt is useful to understand, but it is not a magic SEO file.
It provides a simple way to describe a website and point AI systems toward important resources. The format may become more useful as AI agents interact with websites more often, but support is not universal.
For Google Search, the fundamentals still matter: crawlable pages, useful content, strong site structure, internal links, and clear information.
For AI visibility, go one step further. Track how AI systems actually discover, describe, mention, and cite your brand.
That gives you a much clearer picture than simply checking whether an llms.txt file exists.
You can also explore sourceable to monitor your brand's AI visibility and understand how AI systems represent your business.