What Is llms.txt? Examples, How to Create It, & Its Impact on GEO

More and more users are turning to ChatGPT, Claude, and Perplexity before deciding to purchase a product or use a service. This shift in behavior has forced many business owners and marketing teams to get creative so their content is easier for AI to find, understand, and cite.

In this context, the llms.txt file is a Markdown-based text file placed in the website’s root directory to help AI understand the structure and content of a site, thereby optimizing the website’s visibility to AI.

For a business aiming to succeed in the era of AI search, understanding llms.txt best practices is absolutely essential. This article will thoroughly cover the entire concept of llms.txt, from what it is and its functions to how it differs from robots.txt, how to create it, and how to measure its performance.

Whether you are a business owner, an IT professional, or part of a digital marketing team looking to delve deeper into the concepts of AEO and GEO, be sure to read this article through to the end.

What Is llms.txt?

llms.txt is a Markdown-based file available on a website that provides summaries and links to relevant pages for AI agents. This concept was proposed by Jeremy Howard via llmstxt.org to help AI agents find important information more easily and specifically.

Howard’s idea stems from the fact that modern websites are already equipped with many elements such as navigation, ads, and JavaScript.

Some of these elements can make content more difficult to process. Simply put, the llms.txt file serves as a roadmap so that AI agents can find important information, such as product documentation, policies, or other relevant pages.

However, it is also important to understand that the llms.txt file does not guarantee that the content will definitely be cited by AI. This is because the llms.txt concept itself is still evolving and has not yet become an official web standard.

Moreover, Google has stated that a website does not need to have an llms.txt file to appear in Google Search or AI Overviews. Nevertheless, learning about this concept can help you optimize your website’s SEO strategy as a way to adapt to the new era of search.

Read also: Google AI Overview and Its Struggle to Keep Up with Human Wits

Example of the Correct llms.txt Format

The llms.txt format uses simple Markdown syntax, such as headings to organize information and links to direct users to relevant pages. The following structure serves as a basic example and can be customized to suit the needs of each website.

Obviously, this llms.txt structure does not have to be the same for every website. You can adjust the categories, number of headings, and links based on your business type and the information most relevant to the AI agent, as long as you follow the appropriate Markdown format.

How llms.txt Works

Simply put, llms.txt is designed to serve as a guide for AI agents when searching for information on a website. Here is the general workflow and how llms.txt operates:

• Locating the file: The AI agent accesses the website and locates the available llms.txt file.
• Reading the information: The AI agent reads the summary and the list of links included in the file.
• Navigating pages: From these links, the AI agent can access pages relevant to the information it is searching for.
• Using the information: The content from the relevant pages can then be processed and used for the AI agent’s needs, including when answering user questions.

However, you should also understand that this process may not occur with every AI agent, as support for llms.txt is still limited.

Implementing llms.txt does not guarantee that the file will be automatically recognized by LLM bots. In other words, AI search still largely follows the rules of robots.txt. Therefore, llms.txt should not be viewed as a replacement for robots.txt, but rather as an approach that is still evolving.

Functions and Uses of llms.txt

Although it is not yet an official standard, llms.txt can still be considered as part of website preparation to better adapt to the development of AI search. Here are some of its functions to consider.

1. Identifying the Most Important Content

Creating llms.txt requires you to select the most relevant pages to represent your business. This process can help your team review the content structure and determine which information to prioritize.

2. Making Website Information More Focused

Websites using llms.txt provide a summary and links to pages considered important in a single, simple file. This structure can provide an additional guide for AI agents that support llms.txt when searching for information on a website.

3. Serving as an Early AI Search Experiment

The development of AI search is still ongoing, and not all systems currently use llms.txt. Since its implementation is relatively simple, llms.txt file AI optimization can be used as an additional experiment to prioritize both fundamental and technical SEO.

4. Preparing Your Website for the Era of AI Advancements

It is not yet certain whether AI servers will adopt llms.txt in the future. However, preparing this file can be one way for your business to stay adaptable and keep pace with how AI discovers and processes information from websites.

5. Supplementing Website Information Documentation

llms.txt is not a replacement for robots.txt or sitemap.xml, as all three serve different functions. When used, llms.txt can serve as supplementary documentation that summarizes important information and directs AI agents to relevant pages.

Differences Between llms.txt, robots.txt, and sitemap.xml

The most fundamental difference between llms.txt and robots.txt lies in their purpose. Generally, llms.txt only provides content recommendations without imposing any restrictions; robots.txt controls crawler access permissions; and sitemap.xml helps search engines find URLs to crawl.

Here is a table summarizing the differences between the three in more detail:

These three files are not interchangeable. If a website uses llms.txt, this file is best positioned as a supplement to robots.txt and sitemap.xml, especially for AI systems that support or utilize it.

Can llms.txt Help with GEO & AEO Optimization?

llms.txt can serve as a supporting element for Generative Engine Optimization (GEO) and Answer Engine Optimization (AEO) strategies, as both address how information from a website can be discovered and understood by AI systems.

However, the llms.txt file itself is more of a supplementary element and has not yet been proven to increase the likelihood of content being cited or to serve as a ranking factor in AI systems.

Practically speaking, implementing AEO and GEO still requires content that is relevant to users, a clear information structure, and trustworthy sources.

Therefore, if you implement llms.txt, treat this file as just a small part of the AI search optimization ecosystem, not as a replacement for your existing content strategy and website SEO optimization.

How to Create an llms.txt File

Creating an llms.txt file is essentially quite simple and can be done without a developer’s help if you have access to your website’s files. The following structure can serve as a starting guide and be customized to each website’s specific needs.

• Select the most relevant pages: Identify the pages that best represent your business. Examples include pages about your brand, services, products, documentation, or key reference articles.
• Write the website name and summary: Use an H1 heading to list the brand or website name, then add a summary of the business and the information available on the site.
• Group links by topic: Use H2 headings to organize links into several categories. Include a brief description for each link to make the page context easier to understand.
• Save as llms.txt: Save the file with the name <llms.txt>. If you are using a specific CMS or framework, the location and method for uploading the file may vary, so you will need to adjust it according to your website’s configuration.
• Check file access: After uploading, make sure the file can be accessed via a page such as <yourdomain.com/llms.txt>. You can open it in a browser to ensure the file’s contents display correctly.

How to Check llms.txt

After the file is uploaded, the next step is to ensure the file is properly installed and publicly accessible. Then, how to see the llms.txt file of a website? The simplest way to check this is to open the URL <yourdomain.com/llms.txt> directly in incognito mode in your browser, then verify that plain text appears, not a 404 page or a redirect to the homepage.

Additionally, you can run the `curl` command via the terminal to check the file’s HTTP status and content type, or use a site audit tool to verify whether the `llms.txt` file has been detected in a valid format.

How to Measure llms.txt Performance

Since there are no official metrics yet to measure the effectiveness of llms.txt, its performance can be evaluated using various data available on the website. Some things you can monitor include:

• AI crawler access: Check the server logs to see if bots such as GPTBot, ClaudeBot, or PerplexityBot are accessing the llms.txt file for AI crawlers.
• Traffic from AI search: Compare traffic from AI search before and after the file is implemented to see any changes that have occurred.
• Brand mentions in AI: Monitor whether your brand or website content begins to appear in AI assistant responses such as ChatGPT or Perplexity for relevant queries.
• File content updates: Periodically review the links and information in llms.txt to ensure they remain consistent with the latest content on the website.

However, it is important to understand that changes in traffic or brand appearances in AI assistants cannot be directly attributed to llms.txt. Other factors, such as content quality, source authority, website structure, and changes to the AI system, can also influence these results.

Therefore, it can be concluded that llms.txt is best viewed as one of several experiments in preparing a website for the evolution of AI search. For these efforts to be more comprehensive, its implementation must still be supported by strong fundamental SEO practices, high-quality content, and GEO and AEO optimization tailored to business needs.

If you want to prepare your website to adapt more easily in the AI era, cmlabs can help develop strategies ranging from technical SEO to GEO and AEO.

With an approach tailored to your website’s conditions and business goals, you can build a foundation for search visibility that does not rely solely on a single approach.

To navigate and optimize your business so your website is more easily cited, referenced, and recommended by AI, use AI Search Optimization (GEO/AEO) Services by cmlabs.

Contact us and tell us more about your business needs and challenges.

Read also: What Is Answer Engine Optimization? A Guide to Optimizing Business