Guide · Ai search
How Ai engines choose which sources to cite
Most Ai answers that cite sources come from a retrieval step. The engine searches an index, pulls a short list of candidate pages, then keeps the passages that answer each part of the question from sources it judges reliable. Pages get chosen when the engine can reach them, they match the underlying searches, they answer clearly, and other sources back them up.
Key takeaways
- Each engine searches an index first: its own, Bing’s or Google’s.
- Blocking an engine’s search crawler can remove you from its answers.
- Clear, self-contained passages are easier to select than long, vague copy.
- Corroboration matters: brand mentions, reviews and backlinks make a source easier to trust.
The pipeline behind a cited answer
- Query understanding. The engine rewrites the question into searches, often several (Google calls this query fan-out).
- Retrieval. It pulls candidate pages from a search index.
- Selection. It reads the candidates and keeps passages that directly answer a part of the question, preferring sources that look reliable and current.
- Synthesis. It writes the answer, combining what several sources agree on.
- Citation. It links to some of the sources it used. Not every source used is shown, and not every page shown is equally weighted.
Which index each engine searches
| Engine | Main index it searches | Search crawler to allow |
|---|---|---|
| ChatGPT | Its own crawl plus third-party search providers, including Microsoft | OAI-SearchBot |
| Microsoft Copilot | Bing | bingbot |
| Google Ai Overviews and Ai Mode | Google Search | Googlebot |
| Gemini | Google Search | Googlebot |
| Perplexity | Its own index | PerplexityBot |
| Claude | Web search plus its own crawler | Claude-SearchBot |
Crawler access: search vs training
Ai companies typically run separate crawlers for training models, for building search indexes and for fetching pages when a user asks. They can be allowed or blocked independently in robots.txt. For example, ChatGPT’s document