Generative Engine Optimisation, AI SEO, answer engine optimisation, LLM optimisation: the names multiply faster than the evidence. Under the labels is a real shift. A growing share of searches now end in a generated answer that cites a handful of sources, and the question of whether your page is one of those sources is a new kind of visibility problem. It is also, for the most part, an old kind of problem wearing a new name.
I want to separate the mechanism from the marketing. This guide covers how these systems retrieve pages, what makes content extractable and citable, how to handle AI crawlers in robots.txt, what can and cannot be measured, and where I think the hype is leading people away from work that would actually move the needle. I am testing much of this in the open in the AI search citation test, and this guide reflects what I understand today rather than what I can prove.
How AI answers are built: retrieval, then generation
The single most useful thing to understand is that generative search systems do not answer from memory alone. For anything current, specific or verifiable, they retrieve. Google's AI Overviews and AI Mode draw on the same index and ranking systems as regular search; the language model sits on top, deciding what to synthesise and which retrieved pages to cite. Answer engines that run on other models work the same way with their own retrieval layers, often leaning on a conventional search index as the source of candidate pages.
- User query
- Query fan-out
- Retrieve candidate pages
- Extract passages
- Generate answer
- Cite sources
The step most people miss is fan-out. A single user question gets decomposed into several sub-queries. "Is programmatic SEO worth it for a small SaaS company" might become searches for what programmatic SEO is, when it works, risks for small sites, examples in SaaS, and so on. Each sub-query retrieves its own candidates. Your page is competing not for one query but for whichever sub-queries it answers best. This is why pages that answer one thing clearly tend to be cited over pages that answer everything vaguely.
What makes a page extractable and citable
Retrieval finds candidate pages. Extraction pulls passages from them. Generation decides which passages to use and which to cite. Each stage has preferences, and they are consistent enough to design for.
Clear entities
Language models reason about things: companies, products, people, concepts, places. A page that names its subject unambiguously, defines it early and relates it to other known entities is easier to use than one that gestures at "our solution" and "the industry". This is the practical core of semantic SEO: say what the thing is, say what it is related to, and use the same name consistently across the site and the wider web.
Direct answers near the question
When a system extracts a passage, it favours a self-contained block of text that answers a specific question without needing the surrounding context. A heading that poses the question followed by a two-to-four-sentence answer is the shape that gets lifted. Long preambles, answers split across sections, and "it depends" without saying on what, all reduce extractability.
Structure that survives extraction
Headings that mean something, short paragraphs, tables for comparisons, lists for sequences. Not because the model reads HTML tags for ranking, but because well-structured text chunks cleanly. A comparison buried in a paragraph is harder to extract than the same comparison in a table. The content SEO guide covers this shape in detail; nothing about it is new.
Original data and claims that can be attributed
Generated answers need sources that say something specific. "Our tests showed X" with a method behind it is citable; "experts agree that X" is not, because the model can get the consensus anywhere. This is the same reason original data earns links, which I covered in the link building guide. The two disciplines converge on the same asset.
Brand mentions across the web
Models learn what an entity is from everywhere it appears, not just from its own site. A company that is described consistently on partner sites, in press, in directories, in reviews and on its own pages has a stable identity in the model's view. A company that only describes itself is a weak signal. This is where PR and the long-neglected business of being talked about becomes a search activity.
| Property | Helps citation because | Common failure |
|---|---|---|
| Named, defined subject | The system can confidently map the page to an entity | Pronouns and vague nouns; product never named in the body |
| Question-shaped headings with direct answers | Matches sub-queries from fan-out; chunks cleanly | Clever headings; answers buried after preamble |
| Tables and lists for comparisons | Structured content extracts as a unit | Comparisons written as prose across paragraphs |
| Original data with method | Gives the model something it cannot get elsewhere | Restating consensus; unsourced claims |
| Consistent entity across the web | Reinforces what the brand is and does | Different descriptions on every third-party listing |
| Freshness signals | Many answers favour recent sources for changing topics | Undated pages; stale updates with new dates |
Technical accessibility for AI crawlers
The crawler landscape has fragmented. Googlebot still feeds Google Search and therefore AI Overviews and AI Mode. But there are separate user agents for training data collection and for live retrieval by other AI products, and they do not all behave the same way. Blocking the wrong one can remove you from a product you wanted to appear in; allowing everything hands your content to systems that give nothing back.
I make the decision per bot, per purpose, and I document it in robots.txt with comments so the next person understands why. The categories I distinguish:
Search crawlers
Googlebot, Bingbot and their equivalents. These feed search indexes and, through them, the AI answers built on top. Blocking these removes you from both.
Training-data crawlers
User agents that collect content for model training. Allowing them has no direct visibility benefit; blocking them has no visibility cost in search. A policy decision, not an SEO one.
Live retrieval agents
User agents that fetch pages at answer time for a specific user's question. Blocking these can stop your page being cited in that product even if you rank in conventional search.
Google-Extended and similar controls
Tokens that control use of content for Google's generative models without affecting Search indexing. Worth understanding precisely before touching, because the scope of each token has changed over time.
Whatever the policy, the implementation needs testing. I run every robots.txt change through the robots.txt tester with each relevant user agent before it ships, because a single mis-ordered rule can block Googlebot from a folder while allowing everything else. The underlying rules are covered in the crawlability and indexability guide.
Two more technical points. First, JavaScript-dependent content is a bigger risk here than in conventional search. Several retrieval agents do not render JavaScript at all, so content that only exists after client-side rendering is invisible to them; my JavaScript rendering test covers the general case. Second, structured data remains useful because it states entities and relationships explicitly, though I treat it as a reinforcement of visible content, never a substitute for it.
What you can and cannot measure
This is where I see the most dishonesty, so I will be specific about the limits.
What Search Console shows
Impressions and clicks from AI Overviews are included in the regular performance report but are not broken out separately. You can see total performance; you cannot cleanly isolate the AI portion. AI Mode traffic is similarly blended. Anyone selling a dashboard of "AI Overview clicks" from Search Console data is estimating.
What referral data shows
Some answer engines send identifiable referrers; some strip them; some open links in ways that record as direct. In GA4 I set up a channel grouping for the referrers I can identify, and I treat the result as a floor, not a count.
What citation tracking shows
Running a fixed set of prompts through several AI products on a schedule and recording which domains are cited is possible, and it is what I do in the citation test. The results vary between runs because the answers are non-deterministic. Trends over many runs are meaningful; a single run is not.
What brand metrics show
Brand search volume, direct traffic and branded query impressions are the best available proxy for whether being mentioned in AI answers is reaching people. They lag and they are noisy, but they are real.
What cannot be measured
How often you were considered and not cited. Why a specific answer chose a competitor. The share of a user's decision that an AI answer influenced. Anyone claiming precision here is guessing, and I say so to clients.
Hype versus fundamentals
The generative engine optimisation industry has produced a set of tactics that range from sensible to superstitious. A short triage:
- Sensible: answer questions directly under clear headings; name entities consistently; publish original data; make the site crawlable without JavaScript; earn mentions on third-party sites; keep content current with real updates.
- Unproven but harmless: adding an llms.txt file; extensive FAQ schema; writing summary blocks at the top of pages. None of these hurt. None has strong evidence of helping on its own.
- Counterproductive: stuffing pages with "according to experts" phrasing because a paper suggested citations help; writing for machines in a way that makes the page worse for people; blocking search crawlers out of fear of training use; chasing every new AI product with bespoke content.
The fundamental point, which I made in the note on AI search and fundamentals, is that every item in the sensible column was already good practice. Sites with deep topical coverage, clean technical foundations and a real reputation were cited from the first day AI Overviews launched, without doing anything new. The topical authority work that makes a site the obvious source on a subject in conventional search is the same work that makes it the obvious source for a generated answer.
Where I do change my approach is in emphasis. I now spend more time on entity consistency across the web, more time on making individual passages self-contained, and more time on the robots.txt policy conversation. I spend less time on things that were already marginal, like meta description optimisation, because a generated answer never reads one.
A working approach for Indian businesses
Indian search behaviour adds two specifics. First, a large share of queries are in Hindi, Hinglish or regional languages, and generative answers in those languages draw on a far thinner pool of sources. A company that publishes clearly in Hindi on its subject can become a cited source with much less competition than in English. Second, mobile-first usage means generated answers are often the entire visible screen; the citations beneath them are what remains of organic visibility. Being one of those citations, or not, is a stark difference.
The approach I take is the one that runs through every page on this site: treat it as a system. Technical access first, so the pages can be retrieved. Topical depth second, so the site is an obvious source. Entity hygiene third, so the model knows who you are. Measurement that is honest about its limits. And original work, because that is the only thing that is scarce. The frameworks section documents each of these as a repeatable method, and the AI SEO service is where I do the work for clients who want it implemented.
AI search does not reward a new kind of SEO. It rewards the parts of the old kind that most sites never finished.
Frequently asked questions
What is Generative Engine Optimisation?
Generative Engine Optimisation, or GEO, is the practice of making content more likely to be retrieved, extracted and cited by AI systems that generate answers, such as Google's AI Overviews and AI Mode, and standalone answer engines. In practice it overlaps almost entirely with strong conventional SEO: crawlable pages, clear entities, direct answers, original data and a consistent reputation across the web.
Do AI Overviews use the same ranking as regular Google Search?
They draw on the same index and core ranking systems. A language model then decides what to synthesise and which retrieved pages to cite, often after breaking the query into several sub-queries. Pages that do not rank for anything are not retrieved and cannot be cited, so conventional visibility is a prerequisite.
Should I block AI crawlers in robots.txt?
Decide per bot and per purpose. Blocking search crawlers removes you from search and from the AI answers built on it. Blocking training-data crawlers is a policy choice with no search visibility cost. Blocking live retrieval agents can prevent citation in that product. Test every change with each relevant user agent before deploying.
Can I track traffic from AI Overviews in Search Console?
Not separately. Impressions and clicks from AI Overviews are included in the normal performance report without a dedicated filter. You can run your own citation tracking by querying a fixed prompt set on a schedule and logging cited domains, and you can use brand search growth as a lagging proxy, but precise AI-attributed traffic is not currently available.
Does adding FAQ schema or an llms.txt file get my content cited?
There is no strong evidence that either does on its own. Neither is harmful. Structured data helps state entities explicitly, which is useful, but the visible content has to answer the question clearly first. I treat these as minor reinforcements, not as levers.
Is AI SEO different for Hindi or regional-language content?
The mechanism is the same, but the pool of clearly written sources in Hindi and regional languages is much thinner, so a well-structured page on a specific subject can become a cited source with less competition than the English equivalent. For Indian businesses with a regional audience this is an underused opportunity.
Where this connects
The entity-first thinking here builds on the semantic SEO guide, and the depth that makes a site an obvious source is the subject of my Topical Authority Framework. I am tracking citations in the open in the AI search citation test, and the technical SEO and content architecture case study shows the foundations that have to be in place before any of this applies. If you want it implemented, the AI SEO service covers access, structure, entity work and honest measurement. This guide is the final level of the SEO Roadmap, which is deliberate: everything before it is a prerequisite.