AI companies crawl the web for two distinct purposes: training models, and answering a user’s question right now. Separating those two is a critical decision for a content owner.
Know the crawlers
- GPTBot — OpenAI, model training
- OAI-SearchBot — OpenAI, for ChatGPT search results
- ClaudeBot — Anthropic
- PerplexityBot — Perplexity, live answers
- Google-Extended — Google’s AI training; does not affect normal search
- CCBot — Common Crawl, a data source for many models
An important distinction: blocking Google-Extended does not affect your search rankings, but it limits your use in AI Overviews. Blocking Googlebot removes you from search entirely.
A decision framework
Three scenarios:
- I want visibility: allow search and answer crawlers. This is right for most businesses.
- I do not want to be training data but I do want to appear in answers: block training bots, allow search bots.
- My content is paid or private: block everything and back it with real access control.
robots.txt example (scenario two)
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /What is llms.txt?
llms.txt is a Markdown file at your site root that summarises your site’s structure for language models. robots.txt says "where you may go"; llms.txt says "here is what exists".
Typical contents: what the site does, a list of your most important pages with short descriptions, links to documentation.
To be honest about it: this is a proposed standard, not yet formally supported by the major providers. It costs little to add, so it is reasonable to do — but do not expect a magic effect.
Does blocking work?
Well-behaved crawlers respect robots.txt. Some do not. If you have content that genuinely needs protecting, put it behind authentication — the robots file is a sign, not a lock.
Measurement
Count these bots’ requests in your server logs. Seeing which pages they fetch is the most concrete feedback available on your GEO work. If a page is never fetched, do not expect it to appear in answers.