Different AI crawlers do different jobs, and blocking one does not block the others. robots.txt is a voluntary request with no legal force. llms.txt shows roughly 10% adoption, negligible crawler interest, no measured citation lift, and Google has said it does not support it. Fix rendering and schema before touching either.
The technical side of AEO attracts more confident writing than any other part of it, and a lot of that writing is wrong in a specific way: it treats every AI crawler as the same thing, and it treats llms.txt as established practice.
Neither holds up. So this is the version with the evidence attached, including the parts that argue against doing the fashionable thing.
The crawlers are not one thing
The most consequential misunderstanding in this area is that "AI crawlers" is a single category with a single switch.
They serve at least three distinct purposes, and conflating them leads to decisions that do the opposite of what was intended.
| Purpose | What it does | Consequence of blocking |
|---|---|---|
| Training | Collects content to train future models | You are less likely to be in the model's prior knowledge |
| Search indexing | Builds an index the assistant retrieves from | You cannot be cited in that assistant's answers |
| User-initiated fetch | Retrieves a page because a user asked about it | The assistant cannot read your page on request |
Why the distinction is decisive
Blocking training crawlers while allowing search crawlers is a coherent position: you do not want to be absorbed into a model, but you do want to be citable. Blocking search crawlers while allowing training crawlers is close to the worst of both.
And if the post-hoc citation hypothesis holds, blocking training crawlers has a cost that will not be visible for years, because you are opting out of exactly the prior knowledge that decides whether you get considered.
The main agents, as of writing
Common ones include GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI; ClaudeBot and its search and user-initiated counterparts from Anthropic; PerplexityBot and Perplexity-User; Google-Extended; Applebot-Extended; CCBot for Common Crawl; and Meta's crawler.
User-agent names change and new ones appear regularly. Verify against each provider's official documentation rather than any blog post, this one included. That is not a disclaimer, it is the actual working practice.
The Google-Extended trap
Worth isolating because it is the most commonly misunderstood control in the whole area.
Google-Extended does not control whether you appear in Google Search or in AI Overviews. It governs use of your content for Gemini and Vertex AI model training. AI Overviews are served from the regular search index, crawled by ordinary Googlebot.
So blocking Google-Extended does not remove you from AI Overviews. If your objection is to being summarized in search results, this is not the lever, and the only lever that works is blocking Googlebot, which removes you from Google entirely.
I have seen this decision made incorrectly more than once, by people who read a headline and acted on it.
robots.txt is a request, not a wall
Worth being unromantic about what this file is.
robots.txt is a voluntary convention. Well-behaved crawlers honour it. It carries no legal force by itself and it does not technically prevent access. Anyone determined to take your content will take it.
What it does provide is a clear, documented statement of intent, which matters if you ever need to argue the point, and which the major providers do generally respect.
If you want actual enforcement
You need server-level controls: blocking by user agent or IP at the edge, rate limiting, or authentication. Those are real barriers. robots.txt is a sign, not a lock.
The AI Crawler Decision Matrix
The decision is genuinely situational, so here is how I would reason about it rather than a blanket recommendation.
| Your situation | Training crawlers | Search crawlers |
|---|---|---|
| Content marketing is how you get found | Allow | Allow |
| You sell the content itself (publisher, courses, research) | Consider blocking | Allow |
| Regulatory or confidentiality constraints | Block | Block, and authenticate |
| You have no idea | Allow | Allow |
For nearly every professional services firm, small business, and consultancy, the answer is allow both. Your content is marketing. Its purpose is to be encountered. Blocking the systems that increasingly mediate discovery is choosing invisibility to prevent a theft that costs you nothing.
The genuine exceptions are businesses whose content is the product. A publisher, a research firm, a paid-course business has a real argument, and it is a commercial decision rather than a technical one.
The llms.txt question
Now the fashionable part, and the part where the evidence is weakest.
What it is meant to be
llms.txt is a proposed markdown file at your root that gives AI agents a curated guide to your most important content. The intent is reasonable: help agents find what matters rather than crawling everything.
What the evidence says
Adoption sits at roughly 10.13% in a study of 300,000 domains, after around eighteen months of industry discussion. Earlier data showed about 2.13% adoption with nearly 40% of those files being empty plugin stubs.
Crawler interest has been measured directly, and it is close to nil: 408 llms.txt requests against more than 500 million AI bot visits over a 90-day window. Studies across several datasets show no measurable lift in AI citations from the file alone.
On platform support, Google's Gary Illyes stated in July 2025 that Google does not support llms.txt and is not planning to. Google-Extended and Gemini behaviour does not visibly change in response to it.
The two things people get wrong
It is not access control. robots.txt says what a crawler may fetch. llms.txt suggests what is worth fetching among things already permitted. They are different jobs and llms.txt does not restrict anything.
It does not opt you out of training. Adding the file has no effect on whether your content is used to train a model. Several posts I have read imply otherwise.
My actual recommendation
Do not prioritize it. It costs an hour, it is harmless, and the standard may gain support, so there is a weak case for adding one if you have nothing better to do. There is no case for it ahead of rendering, structure, schema, or off-site presence.
If a provider's AEO proposal leads with llms.txt, that tells you what depth of research went into it.
Schema markup: what it actually does
Structured data has a real role, and it is narrower than the claims made for it.
Schema helps machines resolve what a page is, who published it, and how its parts relate. That supports extraction and entity resolution. It is not a citation switch, and sites with immaculate schema and no authority do not get cited.
The types that earn their place
Organization with complete identity details and sameAs links. Person for authors, with jobTitle and sameAs, applied consistently across everything they write. Article on editorial content. FAQPage where you have a genuine FAQ block. Service on service pages, and BreadcrumbList for hierarchy.
Consistency matters more than coverage. One correct, consistent Organization block sitewide beats six page types implemented differently.
The mistake to avoid
Marking up content that is not visible on the page. It violates guidelines, and more practically, a machine reading both will find a contradiction, which is worse than having no markup at all.
The thing that actually breaks most sites
After all of the above, the most common technical failure I find has nothing to do with AI specifically.
Content that only exists after JavaScript executes. If your main text or your internal links are injected client-side, you are betting on every crawler rendering scripts. Many do not, and the ones that do give it lower priority on low-authority domains.
The test takes thirty seconds. Disable JavaScript, load your most important page, and look at what remains. If the content is gone, nothing else in this article matters yet.
This is an old SEO problem wearing a new name, which is true of most of the AEO technical layer.
Reading a robots.txt properly
Abstract advice about crawler policy is less useful than looking at the file, so here is how I read one and what the common mistakes look like in practice.
The first thing I check is whether there is a blanket disallow that somebody added years ago and forgot. A Disallow: / under a wildcard user agent, left over from a staging environment that went live without the file being updated, is rarer than it used to be and still turns up. It is worth ruling out in ten seconds before analysing anything subtle.
Next I look for whether AI agents are named at all. Most small business robots.txt files do not mention them, which means the agents are permitted by default. That is usually the right outcome, but it is worth knowing it happened by default rather than by decision, because it means nobody has thought about it.
Where agents are named, the mistake I see most often is inconsistency between an agent's crawlers. A site blocks GPTBot, which is the training crawler, while leaving the search and user-initiated agents permitted. That combination is actually coherent if it was deliberate: you are declining to be training data while remaining retrievable and citable. It is almost never deliberate. It is usually someone who read a post about blocking AI, copied one line, and did not realise the others existed.
The reverse error is more damaging. Blocking a search-indexing agent while permitting the training crawler means you have opted out of the thing that produces citations today while remaining in the thing that produces them slowly. If you are going to block, block the training crawlers and keep the retrieval ones, not the other way round.
I also check whether the sitemap directive is present and points to a live, current sitemap, because that is the cheapest discovery aid available and it is frequently stale or missing after a site migration.
The last check is the one people forget: whether anything important is disallowed as a side effect. Directories blocked for administrative reasons that also contain assets the page needs to render, or a blanket disallow on a path that turns out to include a section of the site that matters. Load a key page with a crawler-view tool and confirm what an agent actually receives, rather than assuming the file does what it reads like it does.
None of this takes long. Fifteen minutes with the file and a rendering check will tell you more about your retrievability than any AEO tool will, and it is the layer where a single wrong line silently negates everything else you do.
Frequently asked questions
Should I block AI crawlers to protect my content?
For most businesses, no. Your content is marketing and its job is to be encountered. Blocking is defensible when the content is itself the product, or when confidentiality requires it. Otherwise you are trading discovery for a protection you do not need.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended governs Gemini and Vertex training. AI Overviews are served from the regular search index via Googlebot. Blocking Google-Extended has no effect on AI Overviews.
Is llms.txt worth implementing?
On current evidence, it is optional and low priority. Adoption is around 10%, measured crawler interest is negligible, no citation lift has been demonstrated, and Google has said it does not support it. Add one if you have spare capacity. Do not let it displace real work.
Does schema markup get me cited?
It helps machines understand and attribute your content, which supports citation. It does not cause it. Treat it as necessary hygiene rather than a lever.
How often should I review this layer?
Twice a year for crawler policy, since the agent landscape changes. Schema only when your site structure changes. Rendering should be checked any time you change your site's framework or templates, which is when this usually breaks.
The technical layer of AEO is mostly the technical layer of SEO with a new vocabulary, plus one genuine new decision about crawler access and one fashionable file that the evidence does not currently support.
If you want someone to check which crawlers your site is turning away and whether your content survives without JavaScript, send me your domain. adphconsulting@gmail.com. ADPH is US-facing and delivers its work from the Philippines.