FR Newsletter

AI for Work

Getting cited by AI assistants: what the platforms actually document

Almost everything written about being cited by AI assistants is guesswork. It does not need to be: Google, OpenAI, Anthropic and Perplexity all publish documentation on how their crawlers work and what you control. Read together, it says something simpler than the industry suggests, and it contradicts one popular piece of advice outright.

Three different doors, not one

The first thing the documentation makes clear is that "AI crawler" is not one thing. There are three distinct purposes, and they are controlled separately:

  1. Training: collecting content that may be used to train models.
  2. Search: indexing your pages so they can be surfaced and linked in the assistant's search answers.
  3. User-initiated fetch: visiting your page right now because a user asked a question.

OpenAI states the consequence explicitly: each setting is independent, so a site can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that its content should not be used for training. It adds that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and that it can take around 24 hours from a robots.txt change for its systems to adjust.

Anthropic splits the same way: ClaudeBot for content that may contribute to training, Claude-SearchBot which analyses content to improve search result quality, and Claude-User for sites accessed when a user asks Claude a question. Its documentation spells out the cost of blocking each: disabling Claude-User prevents retrieval of your content in response to a user query, which may reduce visibility in user-directed web search; disabling Claude-SearchBot prevents indexing for search, which may reduce visibility and accuracy in user search results.

Perplexity makes the same distinction. PerplexityBot is for surfacing and linking websites in its search results and is not used to crawl content for foundation models, and it recommends allowing it in robots.txt and permitting requests from its published IP ranges. Perplexity-User supports user actions, and its documentation notes that since a user requested the fetch, this fetcher generally ignores robots.txt rules.

So the first audit is not about content at all. It is reading your own robots.txt and deciding, deliberately, which of these eight or so agents you allow.

Screenshot coming after our hands-on test
A robots.txt audited agent by agent. Screenshot coming with our hands-on tests.

What Google says you need, and what you do not

Google's page on AI features and your website is the most useful document in this field, mostly because of what it refuses to promise.

Its position: existing SEO best practices remain valid for AI Overviews and AI Mode, and no additional requirement and no special optimisation are needed to appear in them. Technically, to appear as a supporting link, a page must be indexed and eligible to be shown in Google Search with a snippet, and meet Search's technical requirements. There are no further technical requirements.

Then the sentence that contradicts a great deal of current advice: you do not need to create machine-readable files, AI text files or markup to appear in these features, and you do not need to add any particular schema.org structured data.

What it does list as good practice is unglamorous and mostly technical:

  • make sure crawling is allowed in robots.txt and by any CDN or hosting infrastructure;
  • make your content easy to reach through internal links;
  • give users a good page experience;
  • ensure important content is available as text;
  • support text with high-quality images and videos where relevant;
  • ensure structured data matches the visible text on the page;
  • keep Merchant Center and Business Profile information up to date.

Two more points worth knowing. Google describes a query fan-out technique, where AI features run several related searches on sub-topics and data sources to build an answer, which is why a wider and more varied set of links can surface than in a classic search. And it says AI Overviews only appear when its systems determine they add value, so they do not appear often.

The most common cause is not your content

Notice that Google's first best practice mentions the CDN or hosting infrastructure, not only robots.txt. Perplexity goes further and documents how to allow its bots through a web application firewall, including a Cloudflare rule combining user agent and IP address conditions, with its published IP endpoints.

In other words: the platforms expect that a meaningful number of sites are invisible because a firewall, a bot-protection rule or a hosting layer is silently blocking these agents. That is far more common than a content problem, and far faster to fix. Check your firewall rules and your server logs before rewriting a single page.

A practical note from the same documentation: Anthropic says blocking IP addresses is not a reliable way to opt out, because it impedes reading your robots.txt, and that its bots support the non-standard Crawl-delay directive. If your concern is server load rather than visibility, that is the lever.

How to measure it honestly

Google states that sites appearing in AI features are included in overall Search traffic in Search Console, in the performance report under the Web search type, and notes that clicks from result pages including AI Overviews are of better quality, with users more likely to spend more time on the site.

For the others, what the documentation gives you is the user-agent strings and published IP address lists for each crawler. Which means the honest measurement is your own server logs: filter on the documented user agents and you can see which of these systems actually fetch your pages, how often, and which pages. No third-party dashboard is needed for that, and no third-party estimate is as reliable.

The audit, in one hour

  1. Open your robots.txt and list what it says about Googlebot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Decide each one on purpose.
  2. Check your firewall and CDN are not blocking the ones you want to allow, using the platforms' published IP ranges.
  3. Check your pages are indexable and allowed to show a snippet. Google lists nosnippet, data-nosnippet, max-snippet and noindex as the controls that limit what is shown; make sure none is on by accident.
  4. Check your important content is text, not an image or a script-only render.
  5. Check your structured data matches the visible text, which is the one structured-data instruction Google gives here.
  6. Read your logs for the documented user agents, and come back in two weeks.

What nobody can promise you

Google says it plainly: even if a page meets all the requirements, best practices and policies, that does not mean Google will crawl, index or serve its content, and indexing and serving are not guaranteed. Anyone selling you a guaranteed place in an AI answer is selling something the platform itself refuses to promise.

We have run no test of our own on citation rates, so this guide contains no figure, no ranking and no claim about which platform cites what. It contains what the four platforms document, which is more than most of what is written on the subject.

Tools in this guide

Questions

Do I need an llms.txt file to appear in AI answers?

Not for Google's AI features. Its documentation states that you do not need to create machine-readable files, AI text files or markup to appear in AI Overviews and AI Mode, and that no particular schema.org structured data is needed. Other systems may use such files, but Google says plainly that it does not require them.

What is the difference between the crawlers of one platform?

They serve different purposes and are controlled independently. OpenAI separates GPTBot for model training from OAI-SearchBot for appearing in ChatGPT search answers. Anthropic separates ClaudeBot for training from Claude-SearchBot for search and Claude-User for user-initiated fetches.

If I block the training crawler, do I disappear from the answers?

Not necessarily, because the settings are independent. OpenAI's documentation gives exactly that example: a site can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that content should not be used for training.

Sources

  1. Google Search Central, AI features and your website
  2. OpenAI, Overview of OpenAI Crawlers
  3. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
  4. Perplexity, Perplexity Crawlers