Inside ChatGPT’s retrieval stack: The index, cache, and pages it actually reads

When ChatGPT cites a web page, where did that page come from?

In July, our team at RESONEO captured and dissected 1,200 ChatGPT answers, 88,000 search results, and 26,900 distinct pages to find out. (Disclosure: I’m the co-founder of RESONEO.)

We found three layers in how ChatGPT grounds its answers: a discovery index that finds pages, a reading cache that keeps full copies of pages it has fetched, and a small set of pages it opens live.

Each layer has its own rules, its own staleness, and its own blind spots. Understanding those layers makes much of ChatGPT’s citation behavior easier to explain — including why some pages are retrieved, some are read, and some ultimately get cited.

How we uncovered ChatGPT’s retrieval system

Our Chrome extension records the raw data stream ChatGPT sends to the browser, including fields the interface never displays. 

Until July 21, that stream contained a field called result_source that named the internal pipeline behind every single search result.

Four values kept coming back: labrador, bright, oxylabs, and serp. None of them is documented by OpenAI, which only speaks publicly of “third-party search providers.”

Then the field vanished from the stream. It happened overnight, on every account we monitor. We now reidentify each pipeline by its formatting signatures (i.e., snippet length, title shape), with a classifier that gets it right about 98% of the time. V5 of our extension ships with that classifier built in. 

We also ran a fleet of accounts across tiers and countries, replayed the same prompts through OpenAI’s API, and published canary pages on our own domain with full server logs, so we could see exactly what OpenAI’s robots fetch rather than guess.

Dig deeper: Inside ChatGPT Search: how web.run and fan-out queries shape AI visibility

The cast of pipelines

What we had established before this round, and what still holds:

• labrador is OpenAI’s own retrieval hub. Besides web search, it returns news, academia (arXiv), Reddit, and YouTube results, so it acts as an orchestrator of in-house indexes, open repositories like Wikipedia, and news partnerships.
• bright and oxylabs are scraped Google results, bought live.
• P1/P2/P3 serve shopping from OpenAI’s own merchant feeds, with Google kept around as an oracle for prices and reviews.
• B1/B3, Yelp, and TripAdvisor serve local results, with a twist: The links displayed to users point to Google Maps.

Shopping and local never touch web search. If you sell products or run a restaurant, your battle is fought in merchant feeds and business listings, on top of the classic citations.

This article focuses on the web search side because that’s where the surprises were.

OpenAI’s web index works differently from Bing

OpenAI has its own web index, and Bing doesn’t supply it. We checked three ways. Only 1.5% of labrador URLs appear in Bing’s top 20 for the same fan-outs.

No labrador snippet matches a Bing snippet. Bing hard-caps titles at 75 characters, while 24% of labrador titles sail past that limit. The longest title we captured runs 289 characters.

That index feeds essentially all search citations in instant mode, the fast mode that free users get. The exception is questions whose answer requires near-live data (“who won last night’s game”), where it splits roughly 50/50 with scraped Google results.

What does a result from that index actually contain? Three things: 

• The URL. 
• The full page title, never truncated, even when it’s absurdly long.
• A snippet of about 200 characters.

The snippet construction deserves a close look because it’s the text from your page that reaches the model in instant mode. It’s anchored on your H1, rendered in caps, plus whatever visible text sits immediately around it. That can include things you wouldn’t necessarily choose: 

• The category label above the title.
• The alt text of the image below it.
• The byline.
• The publication date.
• A table of contents. 

We measured one case where the snippet was 100% table of contents and 0% content. Whatever occupies those 200 characters is what gets picked up, and nothing else. The meta description is ignored entirely.

One more thing about that snippet: It doesn’t depend on the query. The same URL returns the same frozen snippet whether the user asked about pricing, history, or side effects. It’s cut at indexing time and served as is until the next crawl.

By modern search standards, this is pretty crude. Query-independent snippets are the kind of thing web search left behind about two decades ago, and here they are in 2026, powering one of the world’s most-used AI assistants.

We assume this is a placeholder and that it’ll improve fast. For now, it’s what your content gets judged on.

For comparison, bright results behave like Google results: titles are cut around 60 characters with an ellipsis, snippets are around 160 characters, and the meta description is used about one time in three. Your meta description does nothing for labrador and still works for the Google-fed pipelines.

Dig deeper: Bing, not Google, shapes which brands ChatGPT recommends

The routing is economic, not technical

Why does OpenAI maintain both? Cost is the answer.

In instant mode, ChatGPT must answer in a few seconds, and the user typically pays nothing. So it queries only what OpenAI already owns: the labrador hub. No page gets opened (zero pages opened in 93% of instant answers we captured). The response is grounded in titles and 200-character snippets, full stop.

In thinking mode, the user pays a subscription and accepts the wait. Now the expensive tools are invoked: scraped Google rankings through bright (75% of thinking search results) and real page opens performed by the ChatGPT-User robot. A thinking conversation pulls around 100 results across some 28 domains, compared with around 11 results in instant.

Everything that costs OpenAI money (paying scraping providers) or time (opening URLs, calling live APIs) is reserved for the mode where users pay and wait. Here’s why this matters commercially: More than 90% of ChatGPT users are on the free tier. 

Free means instant, and instant means labrador. The pipeline nobody had heard of six months ago is the one deciding visibility in the vast majority of ChatGPT conversations happening right now.

The funnel numbers make the stakes concrete. Across our corpus, 61,332 URLs were surfaced into the sources sidebar. 5,032 became the lead source behind a citation. 759 pages were actually opened, all in thinking mode. An opened page ends up cited 74% of the time. A page merely retrieved but never opened, 7%.

---

A reading cache shared by everyone

This is the part Jérôme Salomon of Oncrawl dug into with us, and it changed how we think about “ChatGPT visited my page.”

Alongside the index, ChatGPT maintains a cache of every page it has ever fetched. Not snippets: full pages, converted from HTML to Markdown, and stored that way. The cache is keyed by URL and shared across all users and tiers. If a free user in Berlin asks about a page that a paying user in Ohio triggered a fetch on last week, Berlin gets Ohio’s copy.

The refresh logic is what engineers call stale-while-revalidate. A copy is considered fresh for about 30 minutes. Within that window, everyone gets the stored version, and your server sees nothing. 

Past 30 minutes, the user still gets the stale copy immediately, and a background fetch refreshes it for the next person. This means the recrawl schedule of your pages is set by one signal only: how often ChatGPT users ask about them. Popular pages stay fresh. Unpopular pages age indefinitely. 

Salomon documented copies served more than 90 days after the fetch, with no eviction cap in sight. Your Cache-Control: no-store header? Ignored. Your noindex? Ignored too.

The Markdown conversion has its own personality. 

• Scripts, iframes, and JSON-LD are stripped (yes, your structured data never reaches the model through this path). 
• Image alt text survives. 
• Text hidden by CSS is still extracted, so the robot reads content human visitors never see. 

We’ll leave the implications of that as an exercise for the reader and note that cloaking detection does not yet appear to be OpenAI’s strong suit.

Two hard limits frame every fetch. The robot does not execute JavaScript, so client-side content is invisible to it. And pages are capped at exactly 4 MB. Past that, the page is not truncated. It is rejected outright with an HTTP 400, and the model reads nothing at all. A page that is too heavy is not partially read. It does not exist.

One surprise hides in all this: an API parameter returns the crawl date of the stored copy of a URL. Fetch your own pages through it, and you learn when a ChatGPT user last caused your page to be read. That is a real, nearly free exposure metric, and nobody is watching it.

The analytics blind spot

You’ve probably seen utm_source=chatgpt.com in your analytics reports. That parameter is appended to the clickable links displayed to users. It tracks outbound clicks.

But not every citation carries that tracking parameter. We observed that the pages ChatGPT opens on its own in thinking mode carry no utm_source in the citations displayed to users.

So the pages the model actually read in full, the ones with a 74% citation rate, leave zero UTM trace. If you gauge your ChatGPT exposure by filtering on that parameter, you’re counting the clicks and missing the reads. Also, watch for the ChatGPT-User user agent in your server logs.

The mystery: Citations that come from nowhere

One finding resists explanation, and we want the community’s help on it.

A share of the results reaching the model carry no snippet at all, just a title and a URL. We see it for arXiv or Reddit results, which come from databases OpenAI has direct access to, so that part is no surprise. But there are many others. 

In instant mode, snippet-less URLs (excluding Reddit, YouTube, and arXiv) get cited more than URLs with snippets (14.9% versus 8.2%). The model prefers citing pages it has read the least about.

Stranger still, some cited links match no search result at all. Our hypothesis is that the model writes them from its parametric memory, the knowledge baked into its weights during training. These are almost always bare domain roots of well-known sites, dutifully suffixed with utm_source=chatgpt.com as if they came from a search. 

A meaningful share of what ChatGPT ends up citing may simply be what it remembers, not what it retrieved. We can observe the phenomenon, but we can’t yet size it precisely, and we would genuinely welcome challengers on this one.

A related oddity: arXiv was pulled more than 2,600 times in our corpus and cited 10 times. Reddit, same pattern. ChatGPT reads preprints and forums to think, then shows the reader ordinary web pages. Being retrieved and being cited are two different markets.

Dig deeper: ChatGPT topic ownership is rare, and SEO alone doesn’t explain it

So, should you write 289-character titles?

The full title reaches the model untruncated, so it’s part of your grounding budget. The 200 characters after your H1 are the only body text the model sees in instant mode.

As such, moving down the category labels, dates, and widgets that sit between your H1 and your first useful sentence directly improves what ChatGPT knows about your page. Both of those are real, measured, and actionable today.

Act on them with a light touch.

• Write the title as a self-contained sentence rather than a truncation-optimized label.
• Make the first 200 characters after the H1 carry your core message.
• Keep an informative alt text near the top.
• Keep your meta description anyway because the Google-fed pipelines still use it.
• Stay under 4 MB and readable without JavaScript.

But don’t rebuild your site around a quirk. Everything in this article describes a system that evolves very fast: the result_source field disappeared overnight, shopping providers were anonymized in the same week, and Google Shopping tokens went from readable to encrypted in four days. Any specific mechanism we documented can be gone by the time you deploy.

Two structural facts, though, are safe to build on. 

• LLMs have knowledge with a cutoff date, past which they know nothing, so they’re condemned to lean on search engines to bridge the gap between their training data and today. Retrieval will remain a permanent dependency of every AI assistant. 
• Every engine they lean on will be pushed toward Google-grade quality because rudimentary indexes produce rudimentary answers, and users notice. The 200-character snippet probably won’t last long. The need to be the best answer will. That’s why the classic fundamentals, crawlable pages, clear titles, and content that actually answers the question keep paying off no matter how the plumbing changes.

The durable battle: Owning the answers to real questions

Optimizing snippet mechanics is a tactic. The strategy is upstream: knowing what people actually ask AI assistants and being the source that answers it.

Nobody asks ChatGPT for “a stroller with a 49 cm frame width.” They ask for “a stroller that fits through metro ticket gates.”

Nobody searches for the foam firmness of headphone ear pads. They ask which headphones stay comfortable when you wear glasses. They ask which sofa survives a cat that scratches.

That gap between spec-sheet language and human language is exactly where AI visibility is decided because assistants answer conversational questions with whatever content maps to them best.

You already own half the corpus that reveals those questions:

• Customer reviews.
• Support conversations.
• The emails you receive every day.
• Surveys nobody reopened.

The other half lives outside your walls, in what people say about your products on forums, on social platforms, in videos, in the press, and on blogs.

Read both sides and the key benefits and pain points of your products emerge in your customers’ own words, which are also the words AI assistants hear.

Mine that language, answer it on pages the robots can read, and you’re optimizing for every retrieval system at once, including the ones OpenAI hasn’t built yet.

What remains open

We don’t know which robot populates the labrador index. Our canary pages caught OAI-SearchBot nearly silent while the index kept serving results.

Reddit is the easy case, since OpenAI has enjoyed real-time access to Reddit’s data API under its 2024 partnership.

YouTube is another story, with no public agreement and no visible fingerprint explaining how those results get in. And the over-citation of snippet-less URLs still rests on a hypothesis.

OpenAI’s stack keeps getting harder to observe. Fields vanish from the stream, providers get anonymized, routing shifts week to week, and the whole thing grows more complex over time.

We published the full study, What ChatGPT pulls, what it shows, what it cites, including the methodology and data behind every number in this article, so others can challenge these findings or push further.