Back to The Lab

llms.txt: Hope, Hype, and Half-Truths

Photo of Idil Woodall
Idil Woodall
SEO Specialist

If you’re like me and spending way more time on LinkedIn than you care to admit, you’ve probably also noticed the changing tone and sensationalism that took over the marketing space.

Yes, we’re no longer writing obituaries for SEO and have accepted that we just need to change with it. Our problem is no longer catastrophising; it has also changed along with SEO – it’s the fauxpertise. 

If you’re not living under a rock, the latest (and if I may say so, the only) hot topic is AI and how we can make it work for us rather than against us. A few months ago, leveraging llms.txt to boost AI visibility emerged as a smart way to do so. 

What is an llms.txt? Why do we care?

The logic is simple – let’s break it down quickly: 

llms.txt is a proposed standard that is meant to help LLMs access and understand content on a given website. It’s meant to direct LLMs on which content to access and not on your website, and it is to be used in conjunction with your robots.txt.

Above is what we know so far; the following is pure speculation: 

If it is an instruction used by LLMs, then technically it could also be used to achieve the following: 

  • Highlight the best pages you’d like to serve to LLMs

  • Improve the chances of being cited by LLMs

There’s also some speculation that llms.txt could help models better understand JavaScript-heavy websites by pointing directly to raw HTML or cleaner content URLs, bypassing rendering issues entirely. 

While unproven, it’s a potentially helpful side effect – especially for sites that struggle with traditional crawler accessibility.

Now here’s where things start to blur.

Somewhere between “llms.txt might be useful” and “AI is everything now”, a wave of SEO professionals started rebranding themselves as AIOs (AI optimisation specialists), GEOs (generative engine optimisers), or other freshly minted titles. The instinct is understandable – we’re all trying to adapt. But there’s a fine line between adaptation and overcompensation.

The issue isn’t ambition; it’s the disconnect between the label and the underlying knowledge. Most marketers pushing AI optimisation as a service aren’t doing anything fundamentally different from traditional on-page SEO or content repurposing. The tech stack may have changed, but the logic hasn’t: understand your audience, serve them value, and make that value discoverable.

The trouble is, there’s now an appetite for quick answers and shortcuts – “add this line to your llms.txt file and get cited by OpenAI” or “optimise your content for Perplexity visibility”. But we’re working with moving targets. At the time of writing, there’s no formal adoption of llms.txt by major LLM providers. OpenAI, Anthropic, and Google have not confirmed how (or if) they would use this standard. We're operating on assumptions layered on top of hope.

What do we do?

That said, it's not all smoke and mirrors. If LLMs do start honouring llms.txt in the future, it could become a strategic touchpoint (one of many) to help shape which content gets surfaced in tools like ChatGPT or AI summaries in search. But until there's clarity, it's speculation dressed up as strategy.

What’s more useful right now is understanding how LLMs work, especially how they access and retrieve information – not paying extra cash to SEO tools for them to tell you whether your llms.txt is up to scratch. 

We're seeing a shift toward retrieval-augmented generation (RAG), where models pull from indexed or proprietary sources to provide answers. Tools like Perplexity, ChatGPT Web Search, and You.com all rely on some form of RAG. 

That means your content doesn’t just need to be indexable – it needs to be linkable, crawlable, and structured clearly enough to be referenced.

Practical tips:

  • Make sure your website architecture is clean and logical. If AI tools use basic crawling heuristics, unclear structures can bury your best content.

  • Use schema markup consistently. Structured data increases the chance of machine-readable relevance.

  • Focus on clarity. LLMs favour well-written, unambiguous content. This is not the time for fluff.

  • Track mentions of your brand or site in LLM outputs. It’s early days, but identifying when and where you’re cited can inform content priorities.

One area that deserves more attention in these conversations is intent. Search engines have long tried to map keywords to user intent: informational, transactional, and navigational. 

LLMs raise the bar: instead of serving a ranked list of links, they attempt to directly resolve intent through synthesis. That makes understanding user needs more important, not less. If your content isn’t aligned with real questions, concerns, or decision-making moments, it won’t surface in a helpful way – whether to a search engine or a generative model. Optimising for AI isn’t just about technical implementation; it’s about precision of purpose.

And let’s be clear – none of this is brand-new. These are the fundamentals of good SEO. If you’ve been focused on crawlability, indexability, structure, clarity, and topical authority, you’re already doing the work that puts you in a strong position for AI visibility. The difference now is the layer of abstraction: instead of ranking on a page, your content might be cited in a conversation. But the underlying signals (clarity, trust, expertise) still matter.

In short: llms.txt is worth keeping an eye on, but it’s not a shortcut. It’s one potential piece in a much bigger puzzle of AI-informed visibility. Real value will come from understanding systems, not slogans.

Related articles