Do You Need an llms.txt File

An llms.txt file is a plain text file written in Markdown that sits at the root of a website and points AI tools towards the pages that explain the organisation most clearly. It has been widely promoted as a quick win for AI visibility, yet the evidence for that is thin and Google has said plainly that its search results ignore the file. Deciding whether to publish one is a small part of AI search readiness for business websites. The right answer comes down to who you expect to read it.

For most B2B organisations an llms.txt file is optional and low risk, but it will not lift rankings or AI citations on its own. The sections below cover what the file is, what Google and the wider industry have said about it, the cases where it earns its place and how to add one to a WordPress website without creating extra work.

What Is an llms.txt File

An llms.txt file is a Markdown document that gives AI agents a short summary of a website and a curated list of links to its most useful content. The format was proposed by Jeremy Howard in September 2024 and is published at llmstxt.org, which describes it as a way to help agents use a website without working through navigation, adverts and scripts.

The proposal sets out a fixed order for the contents so that ordinary software can parse the file as well as language models. Only the first item is required and everything after it is optional.

• An H1 heading with the name of the website or project
• A blockquote with a short summary of what the organisation or project does
• Optional paragraphs or lists with more detail about how to read the linked pages
• Sections under H2 headings, each holding a list of links with a short note on every page
• A section headed Optional for secondary links an agent can skip when it needs a shorter context

The latest version of the proposal goes further and suggests offering a clean Markdown copy of important pages at the same address with .md added to the end. That second step is more work than the llms.txt file itself. Google’s guidance says Markdown copies are not needed for its search features.

How llms.txt Differs From robots.txt and an XML Sitemap

An llms.txt file describes content, whereas robots.txt controls access and an XML sitemap lists pages for indexing. The proposal makes the same distinction, explaining that robots.txt tells automated tools what access is acceptable while llms.txt is read on demand when an agent needs information to help a user.

The practical point is that llms.txt cannot block or allow anything. OpenAI’s documentation on its web crawlers explains that website owners manage how its crawlers treat their content through robots.txt, with separate settings for appearing in ChatGPT search results and for use in model training.

Anyone hoping an llms.txt file will keep content out of AI training or answers needs robots.txt rules instead. Network providers can also block AI crawlers before they reach the server, as covered in Cloudflare’s default block on AI crawlers, so it is worth checking that setting before assuming any AI tool can reach the website at all.

What Google Has Said About llms.txt

Google has said directly that Google Search does not use llms.txt files, including for AI Overviews and AI Mode. Its guide to optimising for generative AI features on Google Search lists llms.txt among the things website owners can ignore, stating that Google Search does not use these files and that creating one will neither harm nor help visibility in its results.

The same guide says it is fine to create and maintain an llms.txt file for other services that use it. Google also notes that it may crawl many kinds of files on a website without treating them in any special way, so seeing Googlebot request the file in server logs is not a sign that it counts.

Google’s search team had taken the same line well before the guide was published. In April 2025 John Mueller compared llms.txt to the keywords meta tag in a Reddit discussion reported by Search Engine Journal, saying that none of the AI services had said they were using it and that server logs showed they did not even check for it.

At Search Central Live in Asia Pacific, Gary Illyes and Amir Taboul described llms.txt as something Google was not looking to adopt. The Search Engine Journal report from the event records that Google sees robots.txt as the main voluntary standard for controlling crawlers.

Why Chrome Lighthouse Checks for llms.txt

Chrome’s Lighthouse tool checks for llms.txt because it is testing how easily AI agents can use a page, which is a different question from search visibility. The Lighthouse documentation for the llms.txt audit describes the file as an emerging convention and says that without it agents may spend more time crawling a website to understand its structure.

The audit only flags a problem when the server returns an error for the file. A missing file that returns a 404 is marked as not applicable, because the documentation treats providing llms.txt as optional for now.

Search Engine Journal pointed out that this leaves Google giving different answers depending on the product, with Search calling the file unnecessary while Lighthouse checks for it in a separate Agentic Browsing category. That category sits apart from the SEO audits, so a passing or failing llms.txt check says nothing about rankings or AI citations.

Do AI Platforms Read llms.txt

The public evidence that AI platforms rely on llms.txt when choosing what to cite is weak, even though the companies behind them publish the files themselves. The proposal notes that OpenAI, Anthropic and Google’s Gemini team publish llms.txt files for their own developer documentation, which shows the format is useful for coding tools that read documentation rather than proving that their assistants use it to judge other websites.

Research by SE Ranking points the same way. Its study of around 300,000 domains found that only about 10 percent had an llms.txt file and that removing the file as a factor made its model of AI citation frequency more accurate, which suggests the file was not explaining why domains were cited.

Your own server logs give the most direct answer for your website. Filtering the logs for requests to /llms.txt shows which user agents fetch the file and how often. Log file analysis for SEO covers how to read those records alongside ordinary crawler activity.

If the file is never requested, it is doing nothing for that website however well it is written. If AI crawlers do fetch it, that still shows interest rather than influence, so the better test remains whether the website is cited in the answers that matter to your buyers.

When an llms.txt File Is Worth Publishing

An llms.txt file is worth publishing when agents or AI tools are likely to need a guided route into detailed content. It is rarely worth much effort for a small brochure website. The cases below are where the format has the clearest use.

The file is a poor investment when it is treated as a ranking tactic. Google has said its search results ignore it, so time spent polishing the file is better spent on the content and technical work that decide whether a page is found and cited.

There is also a maintenance cost to weigh. An llms.txt file that lists retired services or old URLs gives agents an outdated picture of the organisation, so a file nobody looks after can do more harm than having no file at all.

How to Create an llms.txt File on WordPress

Creating an llms.txt file on WordPress takes either a plugin setting or a single text file uploaded to the root of the website. Yoast SEO can generate the file automatically. Its developer documentation for the llms.txt feature explains that the file is created in the root directory and refreshed weekly by a scheduled action.

The same documentation notes that by default Yoast includes the most recently updated posts and pages, with a manual selection option for pages. That default suits a blog more than a B2B website, so choosing the service, sector and about pages by hand usually gives agents a more useful starting point.

Testing is straightforward once the file is live. Opening the address in a browser confirms it loads. Running Lighthouse on the home page then shows whether the agentic browsing check finds the file without a server error.

Websites with several distinct sections may need more than one file, since the proposal allows a file in a subfolder to cover the pages beneath it. That kind of change sits naturally within technical SEO for WordPress websites, alongside robots.txt rules, sitemaps and structured data.

What to Fix Before Worrying About llms.txt

The work that decides AI visibility sits where it always has, in crawlable pages, clear content and consistent information about the organisation. Google’s guidance states that a page must be indexed and eligible to appear with a snippet before it can be shown in its generative AI features, so indexing problems matter far more than a missing llms.txt file.

The technical checklist for AI search readiness covers crawler access, page structure and the signals AI tools use to decide which sources to cite. Structured data belongs on the same list. Structured data for AI explains how schema markup describes an organisation and its content in a form machines can read.

Priority Pixels includes llms.txt in the technical work behind AI SEO, alongside schema, crawler access and the reporting that shows AI referral traffic. Priority Pixels measures AI Share of Voice as a companion metric to traditional rankings, showing where an organisation appears across Copilot, Gemini, ChatGPT and Perplexity.

Whoever does the work, the decision about llms.txt should take minutes rather than weeks. Publish a short and accurate file if agents are likely to need it, then put the real effort into the pages the file points to, because those pages are what AI tools and search engines read in the end.