Do you need an llms.txt file? A sceptical look

TL;DR: llms.txt is a proposal for a markdown file at your site root that gives language models a curated, clean summary of your content. The idea is reasonable and the marketing around it is not, because it is frequently sold as a requirement for AI visibility when the major search provider has explicitly said it does not use it.

Google's own guidance is that llms.txt is not needed for AI Overviews or AI Mode, and it groups the file with special markup and content chunking as things site owners can safely ignore for Search. Where it genuinely helps is a narrower case: developer documentation being ingested into coding assistants and retrieval systems. If that is you, it is worth adding. If your goal is showing up when someone asks an assistant about your business, your effort belongs elsewhere.

What llms.txt proposes

The concept is straightforward and the reasoning behind it is sound.

HTML pages are noisy. Navigation, sidebars, cookie banners, advertising, footers and interactive components surround the actual content, and a system trying to extract meaning has to separate signal from structure. Rendering JavaScript-heavy pages is expensive. Context windows are finite, so wasted tokens have a cost.

llms.txt proposes a markdown file at your site root containing a curated, structured summary: what the site is, what the key sections are, and links to the most important content, ideally in clean markdown form.

The intended benefit is that a language model reading your site gets an accurate, efficient overview instead of parsing a hundred pages of navigation chrome.

Its origin matters for understanding its fit. The proposal came from the developer tooling world, where the use case is concrete: a coding assistant needs to understand a library's API, and pointing it at clean markdown documentation rather than a rendered documentation site is obviously better.

That is a genuinely good idea for that purpose. The question is whether it generalises to a hosting company's blog or a local business site, and the honest answer is that it mostly does not.

What it is not

Several misconceptions worth clearing, because they drive a lot of unnecessary work.

It is not a standard. It is a proposal. Adoption is voluntary, inconsistent, and concentrated in one niche. Nothing obliges any system to read it.

It is not robots.txt for AI. robots.txt controls crawler access and is honoured by major operators. llms.txt is a content summary with no access control function whatsoever. Blocking or permitting crawlers is an entirely separate exercise, covered in the AI crawler guide.

It is not a ranking factor. No search engine treats its presence as a signal. Adding one will not improve your position anywhere.

It is not a requirement for appearing in AI answers. Assistants fetch and read normal web pages. Your content is used or cited based on whether it is relevant and reachable, not on whether you published a summary file.

It is not enforced or verified. Nothing checks whether your llms.txt reflects your actual content, which is one reason systems are cautious about trusting it.

It is not new SEO. The framing of llms.txt as the next essential optimisation is marketing rather than analysis, and it has generated a considerable amount of work that produces no measurable outcome.

What Google has actually said

This is the part that settles the question for most site owners, and it is oddly under-reported relative to the enthusiasm.

Google's guidance on optimising for AI features states plainly that llms.txt is not needed for AI Overviews or AI Mode. It goes further, grouping llms.txt alongside content chunking and special markup as measures site owners can ignore for Search purposes.

The reasoning follows from how these systems work. AI Overviews and AI Mode are built on Google's existing index, which is populated by normal crawling of your actual pages. A separate summary file does not enter that pipeline, and Google has no mechanism to verify it matches your real content, which would make trusting it a poor idea.

The practical consequence: if your goal is appearing in Google's AI features, which is by far the largest AI answer surface for most Indian businesses, llms.txt does nothing. Google has said so directly rather than leaving it ambiguous.

Other operators. Adoption elsewhere is limited and generally undocumented. Some retrieval systems and developer tools read it. Most assistants fetch normal pages. Nobody has published evidence that a business site gained visibility by adding one.

The honest summary: for the surface that matters most, the answer is settled and negative. For other surfaces, the evidence is absent rather than positive.

Where it genuinely helps

The case is real, and it is narrow.

Developer documentation. This is the original and best use. A library, API or framework whose documentation is being read by coding assistants benefits from clean markdown that the assistant can consume efficiently. Several documentation platforms now generate these files automatically for exactly this reason.

If you publish technical documentation that developers use with AI-assisted coding tools, adding one is straightforward and plausibly useful.

Retrieval and internal ingestion. If you or your customers build retrieval systems over your content, a curated index of what matters saves work on the ingestion side. This is a practical engineering benefit rather than a visibility one.

Very large sites with unclear structure. Where the important content is genuinely hard to identify from navigation, a curated index has some value for any system trying to understand the site, including your own tooling.

As a forcing function. Writing an accurate summary of your site's structure and key content is a useful exercise regardless of who reads the file. Several people have reported that the main benefit was discovering how disorganised their own information architecture was.

Low cost. For a documentation site, generating and maintaining one is minimal work. The argument against is not that it is expensive; it is that it is often sold as something it is not.

Where it does not

The cases that cover most businesses reading this.

Small business and service sites. A site with twenty pages does not present a comprehension problem that a summary file solves. An assistant fetching your services page understands it fine.

Blogs and content sites. Individual articles are the unit of value, and assistants fetch them directly when relevant. A summary file does not make a specific article more likely to be cited.

E-commerce. Product data is better expressed through structured data markup, which is actually consumed by search engines, than through a markdown summary that is not.

Anywhere the goal is Google AI visibility. Explicitly not used, per Google's own documentation.

As a substitute for accessible content. If your real problem is that your pages are JavaScript-rendered, slow, or badly structured, a summary file does not fix it. Assistants fetch pages; if those pages are hard to read, that is the problem to solve.

The opportunity cost argument. The hours spent generating and maintaining an llms.txt for a business site could go to writing another substantive article, improving page speed, or fixing structured data. Any of those has better-evidenced returns.

Approach	Consumed by search engines	Effort	Evidence of benefit	Worth it for a business site
llms.txt	No, per Google	Low to medium	Limited, mostly dev docs	Only if you publish developer docs
Structured data markup	Yes	Medium	Strong	Yes
Clean, fast, server-rendered pages	Yes	Medium to high	Strong	Yes
XML sitemap	Yes	Low	Strong	Yes
Well-configured robots.txt	Yes	Low	Strong	Yes
Substantive original content	Yes	High	Strongest	Yes
Content chunking for AI	No, per Google	Medium	None	No
Special AI-specific markup	No, per Google	Medium	None	No

If you decide to add one

For the cases where it makes sense, mainly documentation, here is what a reasonable implementation looks like.

Place it at the site root as /llms.txt, matching the convention.

Write it in markdown. An H1 with the site or project name, a short blockquote summarising what it is, then sections with links to key resources and one-line descriptions of each.

Keep it curated, not exhaustive. The value is in selection. A file listing every page is a sitemap in markdown, which nobody needed.

Link to clean content. Where possible, link to markdown versions of your pages rather than rendered HTML. Some documentation platforms serve a markdown variant at a predictable URL, which is the pattern the proposal envisages.

Keep it accurate. A file describing content that has moved or changed is worse than no file, since anything reading it is misled.

Automate generation if your platform supports it. A hand-maintained file drifts out of date within months, and stale is worse than absent.

Do not expect it to do anything for search visibility. Add it because your documentation consumers benefit, and measure nothing, because there is nothing to measure.

Skip it entirely if you do not publish documentation. This is a legitimate and probably correct decision for most sites.

What to do instead for AI visibility

If your actual goal is being cited when someone asks an assistant about your industry, these have evidence behind them.

Make your pages fetchable and readable. Assistants fetch normal HTML. Server-rendered content, fast responses and clean semantic structure all help. A page that requires heavy JavaScript to show its content is harder for everything to read, as covered in the Core Web Vitals guide.

Allow the search agents. This is the highest-impact single action and it is frequently done backwards. If your robots.txt blocks the agents that fetch pages to answer live queries, you have removed yourself from AI answers entirely. Check this before doing anything else.

Implement structured data markup. Schema for organisation, products, articles, FAQs and reviews is actually consumed by search engines, feeds rich results, and helps machines understand your content. This is the markup that works, as opposed to the AI-specific markup that Google says to ignore.

Answer questions directly. Content structured around specific questions with clear, self-contained answers is more citable than content that requires the surrounding page to make sense. This is why FAQ sections and direct opening paragraphs perform well.

Publish things nobody else has. Original data, real testing, actual numbers from your own operations. Assistants cite sources that contain information they cannot get elsewhere, and this is also what earns links from humans.

Build credible third-party citations. Being referenced by reputable sources is a signal for AI answers as it is for search, and Google's own position is that AEO and GEO remain fundamentally SEO: structured content, technical foundations, credible citations and genuine expertise.

Keep content current. Assistants answering time-sensitive questions favour recent, accurate sources.

None of this is new, which is the point. The measures that work for AI visibility are the measures that already worked.

How to think about proposals like this

A useful frame for the next one, because there will be one.

Ask who adopted it. A proposal with implementation by the platforms that matter is different from one with enthusiasm from consultants. Check for documentation from the actual operators.

Ask what the mechanism is. How exactly would this file change what an assistant does? If the answer requires the assistant to look for something it has no reason to look for, be sceptical.

Ask what the operators say. Google publishes guidance on optimising for its AI features. Reading it directly is more useful than reading commentary about it, and in this case it answers the question explicitly.

Ask what it replaces. If a proposal duplicates something that already works and is already consumed, the new thing needs to be better rather than merely newer.

Ask about the incentive. A lot of AI-optimisation content exists because it is a good hook for selling services, not because the measures are evidenced.

Watch the cost. Low-effort measures with uncertain benefit are fine to adopt speculatively. High-effort ones need better justification than "it might matter later."

Revisit periodically. This is a genuinely moving area, and a position that is correct today may not be in a year. The answer here could change if major operators adopt the file, and it has not so far.

FAQs

What is llms.txt?

A proposed markdown file at your site root giving language models a curated summary of your content: what the site is, its key sections, and links to important pages in clean form. The reasoning is that HTML pages contain substantial noise, so a structured summary is more efficient to read than parsing rendered pages.

Does Google use llms.txt?

No. Google's guidance on optimising for AI features states that llms.txt is not needed for AI Overviews or AI Mode, and groups it with content chunking and special markup as measures site owners can ignore for Search. AI Overviews are built on Google's existing index, which comes from crawling your actual pages.

Is llms.txt a ranking factor?

No. No search engine treats its presence as a signal, and adding one will not improve your position anywhere. It is not consumed by ranking systems, and the framing of it as the next essential optimisation is marketing rather than analysis.

Is llms.txt the same as robots.txt?

No. robots.txt controls crawler access and is honoured by major operators as an access preference. llms.txt is a content summary with no access control function at all. Deciding which crawlers may fetch your site is an entirely separate exercise from publishing a summary file.

Who should actually add an llms.txt file?

Primarily anyone publishing developer documentation, since coding assistants and retrieval systems benefit from clean markdown describing an API or library, and several documentation platforms generate it automatically. It is also useful if you or your customers build retrieval systems over your content.

Will llms.txt get me cited in AI answers?

There is no evidence that it does for a business site. Assistants fetch and read normal web pages, and your content is used based on relevance and reachability rather than on whether you published a summary file. The highest-impact action for AI citation is ensuring search agents are not blocked in robots.txt.

Is llms.txt a web standard?

No, it is a proposal with voluntary and inconsistent adoption concentrated in developer tooling. Nothing obliges any system to read it, nothing verifies that it matches your actual content, and the major search provider has stated it does not use it.

What should I do instead for AI visibility?

Allow AI search agents in robots.txt, which is the single highest-impact action and frequently done backwards. Then make pages fast and server-rendered, implement structured data markup, answer specific questions directly in self-contained sections, publish original data nobody else has, and build credible third-party citations.

Does adding llms.txt hurt anything?

No, beyond the time spent. The argument against is opportunity cost rather than harm: those hours would produce better-evidenced returns spent on page speed, structured data or another substantive article. A stale file describing content that has moved is mildly worse than none, since anything reading it is misled.

How is llms.txt different from a sitemap?

A sitemap is an XML list of URLs consumed by search engines to aid crawling, and it demonstrably works. llms.txt is a curated markdown summary intended for language models, with limited adoption and no consumption by search engines. A file listing every page defeats the purpose, since the intended value is in selection.

Should I add llms.txt to my WordPress site?

If you publish developer documentation, possibly. For a typical business site, blog or store, no. A twenty-page site does not present a comprehension problem the file solves, and product data is better expressed through structured data markup that search engines actually consume.

Could this change in future?

Yes, and it is worth revisiting periodically. If major operators begin consuming the file and publish documentation saying so, the calculation changes. As things stand, the largest AI answer surface has explicitly said it does not use it, and adoption elsewhere is limited and undocumented.

Conclusion

The idea behind llms.txt is reasonable. HTML is noisy, curated markdown is cleaner, and for developer documentation being read by coding assistants that is a genuine improvement worth implementing.

The problem is the gap between that narrow, sensible case and how the file is marketed to everyone else. The largest AI answer surface has stated directly that it does not use llms.txt, and lists it alongside chunking and special markup as things to ignore. That is unusually clear guidance and it settles the question for most site owners.

So the honest position: add it if you publish developer documentation and your platform can generate it automatically. Skip it otherwise, and do not feel behind for skipping it.

If AI visibility is the actual goal, the highest-impact thing you can do today takes five minutes: check that your robots.txt is not blocking the search agents that fetch pages to answer live queries. A surprising number of sites have blocked themselves out of AI answers while trying to opt out of training, and no summary file compensates for that.

After that, it is the familiar list — fast server-rendered pages, structured data that search engines actually consume, direct answers to specific questions, original information nobody else has, and citations from credible sources. Which is to say: the things that already worked continue to work, and that is not an accident.

HostCloud runs LiteSpeed with server-level caching, HTTP/3 and NVMe storage on Indian infrastructure, which addresses the part of AI visibility that is actually mechanical: fast, reliably fetchable pages. Plans from ₹99 a month at https://hostcloud.in.