llms.txt: What It Is and Whether It Helps AI Search Visibility

An llms.txt file is a proposed Markdown file, placed at a website’s root, that gives large language models a short summary of the site and a curated list of links to its most useful pages. As of October 2026, Google Search says it ignores llms.txt, and no major AI search provider has said it uses the file to rank or cite websites. It costs little to publish, but it isn’t a visibility lever.

The file has spread quickly because it is cheap to create and several SEO plugins now generate it automatically. Adoption has run ahead of evidence, and the public statements from search and AI providers point in a more cautious direction than much of the advice online.

What Is llms.txt?

llms.txt is a plain Markdown file that lists a website’s key content for AI systems, in a format that is easier for a language model to read than a full HTML page. It was proposed by Jeremy Howard in September 2024 and is documented at llmstxt.org.

According to the llmstxt.org proposal (2024, revised 2026), the only required element is an H1 with the name of the site or project. A short blockquote summary, optional body text, and H2 sections containing lists of Markdown links can follow, along with an “Optional” section for secondary links an AI agent can skip when it needs less context.

The proposal also recommends offering clean Markdown versions of important pages at the same URL with “.md” added. The idea is that an AI tool working within a limited context window can read a compact summary instead of parsing menus, scripts and layout code.

A minimal file for a service business might look like this:

# Example Agency
> Cross-border branding and digital marketing agency with offices in Singapore, Tokyo and New York.
## Services
- [Web Design and Development](https://example.com/web-design-development/): Multi-language WordPress and Shopify builds
- [Search Engine Marketing](https://example.com/search-engine-marketing/): SEO and paid search across markets
## Optional
- [Case Studies](https://example.com/case-studies/)

How Is llms.txt Different From robots.txt and sitemap.xml?

llms.txt is a content guide, while robots.txt is an access control file and sitemap.xml is a discovery file for search engines. The three files do different jobs, and llms.txt doesn’t replace either of the others.

Access for AI crawlers is still governed by robots.txt. OpenAI’s crawler documentation explains that GPTBot, OAI-SearchBot and ChatGPT-User are each controlled through separate robots.txt rules, and Anthropic‘s help center describes the same approach for ClaudeBot, Claude-User and Claude-SearchBot. Blocking or allowing these bots in llms.txt has no effect.

Does Google Use llms.txt?

Google Search does not use llms.txt, including for AI Overviews and AI Mode. Google’s own documentation states that publishing the file will neither help nor harm a site’s visibility in Search.

According to Google Search Central (2026), site owners don’t need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, “as Google Search itself doesn’t use them.” The same guide says it is fine to maintain llms.txt for other services that use it, but that Google Search ignores the file.

This position is consistent with earlier comments from Google staff. In April 2025, Google’s John Mueller wrote on Reddit that, to his knowledge, none of the AI services had said they were using llms.txt, and he compared the file to the keywords meta tag.

Why does a Google tool check for llms.txt, then?

Chrome’s Lighthouse tool includes an llms.txt audit, but it relates to AI agents browsing a site, not to Google Search rankings. According to Chrome for Developers (2026), the audit only flags a page when the server returns an error while fetching the file, and a missing file is marked as not applicable.

The two positions aren’t contradictory, because Lighthouse checks whether AI agents can use a site, while the Search guidance covers ranking.

Do ChatGPT, Claude and Perplexity Use llms.txt?

None of the major AI search providers has published documentation saying llms.txt affects whether a site is retrieved or cited in their answers. Their published crawler documentation focuses on robots.txt and named user agents instead.

The OpenAI crawler overview explains that OAI-SearchBot surfaces websites in ChatGPT search and that each bot is controlled independently through robots.txt. It does not describe llms.txt as a signal for search inclusion.

Both OpenAI and Anthropic do publish llms.txt files for their own developer documentation. That use case, helping coding assistants and developers find the right API page, is the one the proposal was originally designed for, and it differs from ranking marketing websites in AI answers.

What Does the Usage Data Show?

The usage data shows wide adoption and very little readership. Most llms.txt files receive no requests at all, and only a minority of the requests they do receive come from AI systems.

According to a study by Ahrefs (2026) of 137,210 domains, 28% published an llms.txt file, but 97% of those files received zero requests in May 2026. AI bots accounted for 19.5% of requests to llms.txt files, and the authors note that a fetch doesn’t prove the content was used.

The same study found that no AI bots requested llms.txt paths that returned a 404, which suggests AI systems aren’t actively looking for the file across the web. Ahrefs notes the 28% figure is likely an upper bound, since its customers skew technical.

Should a Brand Add an llms.txt File?

A brand can add an llms.txt file if it costs almost nothing to maintain, but it shouldn’t expect any change in Google rankings or AI citations from doing so. The effort is better spent on the site structure that search and AI systems are documented to read.

The decision depends on the type of site and how the file would be created:

Automatic generation needs a quick review. One widely used WordPress SEO plugin, for example, builds its file from a small set of recently modified posts and pages and refreshes it weekly, so the output may not reflect a brand’s most important service pages.

What Matters More Than llms.txt for AI Search Visibility?

Crawl access, server-rendered content, clear page structure and accurate structured data matter more than llms.txt for AI search visibility. These are the factors that Google, OpenAI and Anthropic actually document.

1. Allow the right crawlers. Check robots.txt rules for OAI-SearchBot, Claude-SearchBot and Googlebot separately, since blocking a training crawler and blocking a search crawler have different effects.
2. Render content in HTML. Key copy, prices and service descriptions should be present in the page source rather than loaded only through JavaScript.
3. Lead each section with the answer. Pages that answer a question in the opening sentences are easier for both search snippets and AI summaries to extract.
4. Keep entity information consistent. Company name, offices, services and contact details should match across every market version of the site.
5. Use structured data where Google supports it. Google’s guide says no special schema is needed for AI features, but standard markup still supports rich result eligibility.

For multi-market brands, these factors apply to every language version of the site. In the multi-language WordPress and Shopify builds we scope at Creative For More, the more common problem is inconsistent structure between market versions rather than a missing AI text file. Our international website development guide covers how to structure market subdirectories and language versions so they are crawled and understood consistently.

Is llms.txt Likely to Become a Standard?

llms.txt may still become more widely used, but in October 2026 it remains a community proposal without formal support from Google Search or a published commitment from the major AI search providers. Its strongest current use is in developer documentation.

An AI provider could announce support at any time, and a brand that publishes the file now loses very little if it keeps the file accurate. We treat it the same way when building sites, and we focus our generative engine optimisation work on the structural factors above.

Brands planning a new site or a rebuild can get AI-search-ready structure built in from the start through our web design and development service, which covers multi-language architecture, international SEO and content structure across Singapore, Japan and the US.

If you’re planning a website that needs to be found by both search engines and AI assistants across several markets, we can review your current structure and set priorities. If your company is looking for a strategic marketing partner, the Creative For More team can help. Book a discovery call to explore how we can support your brand’s growth.

Read More

• International Website Development: The 2026 Guide for Multi-Market Brands
• The AI Stack We Use (And Why We Built It This Way)
• Our Guide to Google’s Latest Search Algorithm Update
• SEO Checklist