What Did the AI Actually Read?

It is simple, really; and at this point, it is almost a habit. We upload a document, write a prompt, and ask an AI system to help us do something with it. Maybe it is a report at work, a policy document, a research paper, a spreadsheet, or some other material connected to the problem or project in front of us. We give the system the file, tell it what we need, and expect it to use what we provided.

Most of the time, we probably do not think very hard about what happened between uploading the document and receiving the answer. We say the AI read it. We say it analyzed the report, and we say it looked through the file and found what mattered. But what exactly do we mean when we say that?

Did the model actually receive the entire document? Did the application divide it into smaller sections and search for the parts most closely related to our prompt? Did it examine the paragraphs around the information it retrieved? Did it understand where that information sat within the larger argument of the document? And if the answer includes a citation, does that citation actually establish that the claim being made is supported by the source? These questions sound technical; however, the distinction matters the moment we begin relying on the answer.

The document is not necessarily what the model sees

Part of the confusion begins with the word document. We tend to treat the file we uploaded and the information available to the model as though they are the same thing. They may not be. There is no single process that necessarily occurs when someone uploads a document to an AI system. Different systems can handle the same action very differently. Some applications use retrieval systems that divide documents into smaller passages, create mathematical representations of those passages, index them, and search the resulting collection when a user asks a question. OpenAI, for example, documents a retrieval architecture in which files placed into a vector store are chunked, embedded, and indexed for later search.¹

Other systems can use materially different approaches. Google's documentation for Gemini describes PDF processing in which textual and visual information from a document can be provided as document context rather than relying exclusively on a conventional retrieval pipeline.² The point is not that one approach is inherently better than the other. The point is that the ordinary statement I uploaded the document tells us very little about what eventually reached the model. And this becomes especially important when retrieval is involved.

A source document contains words, paragraphs, headings, tables, sequence, qualifications, footnotes, argument structure, and relationships between sections. An embedding is something different. Dense retrieval systems can encode passages and questions into vectors and use similarity between those mathematical representations to identify passages likely to be relevant to a question.³ The embedding is not a smaller mathematical copy of the document in the ordinary sense. It is useful because it helps the system navigate toward information that appears related to what we asked, a semantic approximation. This distinction may sound obvious, but I think we routinely forget its consequence. The source is the source, and its representation is a way of finding something inside it. These are not the same thing, nor are they, from my standpoint, interchangeable.

If a retrieval system identifies five passages as highly relevant to my question, those passages may be excellent places to begin looking. But finding the most relevant part of a document is not necessarily the same thing as understanding what the document says. Retrieval can help answer the question: "where should I look?" Research has to answer another question: "what else do I need to understand before I can responsibly use what I found?"

Relevant information is not necessarily sufficient information

Imagine a research paper contains a paragraph saying: "Earlier studies suggested that X might improve the outcome."

If I ask an AI whether research has found that X improves the outcome, that sentence is obviously relevant, and a retrieval system could reasonably identify it as a strong match. Now imagine that the next paragraph says those earlier studies relied on unusually narrow populations and that later experiments failed to reproduce the effect. And then additionally, the conclusion of the paper states the authors do not consider the evidence sufficient to establish X. Nothing about the first paragraph was false, nor did the retrieval system necessarily fail by identifying it. The passage was, in fact, relevant to the question and the language model could accurately summarize the passage it received. It could cite the correct paper and even avoid fabricating a single fact. And yet the answer, "Research has suggested that X improves the outcome", could still leave the reader with a distorted understanding of what the paper actually establishes.

This is a different problem from hallucination. No source had to be invented, nor quotation fabricated. No citation had to point to the wrong paper, as everything visible on the surface of the answer genuinely looks healthy. The problem is that semantic relevance and evidentiary sufficiency are not necessarily the same thing.

Research on retrieval systems increasingly recognizes the importance of this problem. Work on document-structure-aware retrieval notes that conventional retrieval-augmented generation (RAG) systems often treat retrieved passages as isolated chunks and may order those passages by relevance rather than preserving their original document sequence. The authors show that incorporating document hierarchy and structural relationships can improve performance in the settings they evaluated.⁴ This matters because documents are not simply collections of interchangeable paragraphs. Authors build arguments, introduce a question, describe previous work, establish assumptions, report findings, qualify those findings, discuss limitations, respond to contradictions, and eventually reach some broader conclusion. Nothing is in a vacuum and the meaning of one paragraph may depend heavily on what came before it or what follows several pages later. All of this is to say, a retrieved passage can therefore be faithful to the text and still be an incomplete representation of the source. And this distinction matters: Passage fidelity is not necessarily document fidelity.

A system can correctly tell me what one passage says without correctly telling me what the document says, and real retrieval environments exacerbate the problem. Research examining naturally retrieved evidence has found that retrieved material can differ not only in relevance, but also in sufficiency, reliability, interpretability, and stance. Evidence can support a claim, refute it, remain neutral, or simply be insufficient to establish anything meaningful.⁵ That seems obvious when stated directly, yet most people interacting with an AI system never see that layer. We see the answer. We rarely see the competing passages that were not selected, the neighboring paragraphs that were not included, the sections that qualified the retrieved statement, or the evidence that happened to rank slightly below whatever threshold determined what reached the model. That does not mean the system ignored those things intentionally. In many cases, the system may never have encountered them at all.

More context does not necessarily solve the problem

A reasonable response to all of this might be to simply give the model the entire document. Context windows have become substantially larger, and if the problem is that retrieval may exclude information necessary to understand a source, then perhaps the answer is to avoid that selection process altogether. That certainly changes the problem, but it does not necessarily solve it. Information being present in a model's context is not the same thing as that information being used reliably in producing an answer. Research on long-context language models has repeatedly demonstrated that the amount of information a model can technically receive and its ability to consistently use all of that information are different questions. Earlier work found that the position of relevant information within long contexts could materially affect performance in the models evaluated, even though the information itself remained available to the model.⁶ More recent research complicates the exact shape of that effect, as newer models have improved in some long-context settings, but the larger distinction remains: the ability to place information inside a context window does not establish that the information will be uniformly or completely incorporated into the resulting reasoning. To put it simply, this is a question of salience: the user can influence what the model is given, but not fully determine what the model will treat as important once that information is there.

This becomes even more apparent when a task requires the model to work across a large amount of the information it has been given rather than locating one particularly useful passage. Research evaluating models on high-information-coverage tasks has found meaningful degradation when successful completion requires broader integration of the available context.⁷ Other controlled work has found that performance can worsen as context grows even when the relevant evidence can be retrieved successfully, suggesting that finding information and reasoning effectively with that information remain separate problems.⁸ Again, this does not mean that long-context models cannot reason over documents or that giving a model more information is somehow undesirable. It means that we should probably resist another convenient collapse. Information can be present without being meaningfully incorporated into an answer, just as information can exist in a database without ever being retrieved from it.

This also gives us a more useful way of thinking about failure. If an AI produces a poor answer about a document, it is tempting to say the model got it wrong. But where did the failure actually occur? Did the retrieval system fail to locate necessary evidence? Did it retrieve the right evidence alongside distracting information? Was the necessary information visible to the model but poorly integrated into its reasoning? Did the model interpret good evidence incorrectly? Those are materially different failures even though the user experiences all of them as one disappointing answer.

Research frameworks designed to evaluate RAG increasingly separate these stages for exactly that reason. RAGChecker, for example, evaluates retrieval quality separately from generator behavior, including whether the resulting answer is faithful to the provided context, whether relevant information is actually used, and how the model behaves when useful evidence appears alongside noise.⁹ Research examining hallucination in retrieval-augmented systems similarly demonstrates that providing external evidence does not automatically prevent unsupported or contradictory claims from appearing in generated responses.¹⁰ The system, then, is doing more than simply "reading a document." Information has to move through several stages before it becomes an answer, and success at one stage does not guarantee success at the next. The document may be available. The appropriate passage may be retrieved. The passage may be visible to the model, and the model may still fail to use it appropriately. None of these distinctions make the system useless, but they do make the word read feel increasingly inadequate for describing what actually occurred.

Then there is the citation

This is where I think the problem becomes particularly easy to overlook, because citations carry a kind of authority that most of us have been trained to recognize long before generative AI existed. We see a statement followed by a little number, scroll to the bottom of the page, find a legitimate paper or report, and something about the claim immediately feels more grounded.

This instinct is not unreasonable, as citations are fundamental to research because they allow someone else to inspect the source from which a claim was derived. But a citation is also a relationship between a proposition and evidence, not merely the presence of a source somewhere nearby. And this distinction becomes important when AI generates that relationship for us.

Research into citation-generating language models does not simply ask whether an answer contains citations. Researchers separately evaluate whether those citations actually support the claims to which they are attached and whether important claims receive adequate support at all.¹¹ The distinction exists because attaching a citation to a sentence does not itself establish that the sentence is justified by the cited material.

A 2023 evaluation of four generative search systems makes this particularly uncomfortable. In that study, researchers found that 51.5% of generated sentences were fully supported by citations and that 74.5% of the citations they evaluated supported the sentences to which they were attached.¹² Those numbers describe particular systems evaluated in 2023, and it would be inappropriate to turn them into a claim that modern AI citations are wrong half of the time. The technology has changed, the evaluated systems were specific, and the measurements themselves answer particular questions, but the broader distinction does not disappear simply because the systems improve.

A citation can point to a real source without that source supporting everything in the sentence. It can support part of a claim while leaving another part unsupported. It can point toward a genuinely relevant passage while missing a qualification elsewhere in the source. And perhaps most interestingly, a citation that correctly identifies a supporting source does not necessarily tell us whether that source was actually responsible for the answer the model produced. This last distinction is still an emerging area of research and deserves some care. Recent mechanistic work has examined whether citation behavior and answer generation can be altered somewhat independently in a controlled retrieval-augmented setting.¹³ The finding does not justify saying that AI systems simply write an answer and randomly attach citations afterward. However, it does raise a separate question that I think matters more: is a citation merely correct, or is it faithful to the information pathway that actually produced the answer?

Most users will never ask that question. They will see a statement followed by a citation and reasonably assume that the system found the source, read the relevant evidence, used that evidence to form the conclusion, and attached the citation to show them where the conclusion came from. But those are several different propositions hiding behind one superscript. And once again, everything can look perfectly healthy on the surface.

An AI does not have to hallucinate to mislead us

Hallucination receives an enormous amount of attention in conversations about generative AI, and for good reason. Models can fabricate facts, quotations, sources, people, events, and explanations. Those failures matter, and learning to recognize them should remain part of using these systems responsibly. But I increasingly think hallucination may be the easier epistemic problem to understand.

If a citation points to a paper that does not exist, something clearly failed. If a quotation cannot be found anywhere in the source, we can identify the fabrication. If the model invents a date or attributes a statement to the wrong person, there is at least a reasonably clear proposition that can be checked against reality. The more complicated problem is what happens when almost everything is real?

The paper exists, the retrieved passage exists, the passage is relevant, the model accurately describes what that passage says and the citation even points to the correct source. Nothing needs to be fabricated, and yet the resulting interpretation can still be misleading because something necessary for understanding the evidence never became part of the answer. Maybe the neighboring paragraph contained an important qualification. Maybe the section was describing earlier research the authors later challenged. Maybe the methodology only supports a much narrower conclusion than the answer implies. Maybe the conclusion explicitly warns against the interpretation the model eventually produced. Or perhaps another relevant source offered contradictory evidence but ranked just below whatever threshold determined which information was retrieved.

Nothing in that chain necessarily requires the system to lie. And this matters because we often evaluate an AI answer by looking for obvious signs of failure. Is the source real? Yes. Does the cited page contain something related to the claim? Yes. Does the answer sound coherent? Yes. Does it contain appropriate caveats? Perhaps. At some point, the accumulation of those signals begins to feel like validation. But an answer can contain no fabricated information and still leave us with a distorted understanding of the evidence.

This is not uniquely an AI problem, either. Human beings have been cherry-picking evidence, quoting people out of context, overstating research findings, ignoring methodological limitations, and citing papers they barely read for a very long time. AI did not invent bad research practices. What AI changes is the speed and scale at which an interpretation can be assembled, along with how little of the intermediate process the person receiving the answer may ever see.

A researcher who spends an afternoon reading papers can still reach a poor conclusion. An AI system can search, retrieve, summarize, synthesize, and present an apparently well-supported conclusion in seconds. That capability is extraordinary, but it also means that many of the intellectual decisions once made visibly by a researcher can now occur somewhere between the prompt and the answer. If we cannot see those decisions, then the polished output becomes very easy to mistake for the process that produced it.

Retrieval is not the same thing as research

None of this is an argument that retrieval is bad or that long-context models and AI-generated citations are useless. Retrieval is incredibly useful precisely because no person can manually search every report, paper, database, or archive relevant to a question. It helps us navigate information spaces that would otherwise be practically inaccessible.

But navigation and research are not the same thing. Retrieval can tell us where to look; research asks whether what we found is sufficient for what we are trying to say. Sometimes a paragraph is enough. Sometimes the methodology, limitations, surrounding section, or another source changes what the evidence can reasonably support. There is no magical number of neighboring chunks or universal context window that guarantees understanding. The amount of evidence required depends on the proposition being made. Relevant information is useful, but relevance alone does not make it sufficient.

We should probably hold ourselves to the same standard

There is an obvious problem with writing an essay criticizing AI systems for retrieving convenient evidence without considering the surrounding source: I could do exactly the same thing while writing this essay. I could search for papers on retrieval, context windows, document structure, and citations. I could find sentences that sound supportive of the argument I already wanted to make, ask an AI to summarize them, attach citations, and call the result researched. It would be remarkably easy, and it would also reproduce the exact problem I am describing.

So while researching this piece, I found myself asking a slightly different question of the sources. Not simply, "does this paper contain something that supports my argument?" but, "what does this paper actually allow me to say?" That requires looking beyond the convenient passage. What did the researchers test? Which models, datasets, or conditions produced the result? What did the authors identify as a limitation? Is the sentence I want to cite describing the authors' own finding or previous work they are discussing? Does the conclusion strengthen the claim I want to make or narrow it? Is there evidence elsewhere in the paper that changes how the relevant passage should be understood? This does not mean every person has to read every source from the first page to the last before using AI for research. That standard would make many of the benefits of these systems disappear. Human researchers have always used search, indexes, abstracts, references, and other mechanisms to decide where deeper inspection is necessary.

The harder discipline is calibration. How much of this source do I need to inspect before making this particular claim? How consequential is the claim? What kind of error would matter? What would change my interpretation? Which part of the statement is directly supported by evidence and which part is my own inference? Perhaps AI-assisted research should make those questions easier to answer rather than making them easier to skip.

So what did the AI actually read?

The more I think about it, the more I suspect that the word read is doing far too much work. When an AI system tells me it analyzed a document, I increasingly want to know what that means within the system I am actually using. What representation of the source reached the model? Was the document provided directly, or did another component retrieve portions of it? If only portions were retrieved, how were they selected? What surrounding context was retained? What evidence was available but not selected? What information reached the model, and what information actually appears to have influenced the answer? And then there is the question that probably matters most: does the evidence available to me justify the conclusion I am being asked to accept?

These questions are not an argument against using AI to understand complicated information. If anything, they come from taking the capability seriously. We are building systems capable of helping people navigate bodies of knowledge that no individual could reasonably inspect alone. They can help us find information buried in hundreds of pages, compare documents in seconds, connect ideas across sources, and make complicated material substantially easier to work with. There is something genuinely beautiful about that.

But making information easier to find does not eliminate the need to understand what the information means. Access is not consideration. Retrieval is not understanding. Relevance is not necessarily sufficiency. A citation is not automatically justification. And a polished answer is not evidence that every important part of the process behind it occurred the way we assumed.

The next time an AI tells us it reviewed the document, perhaps the most useful question is not whether the file was uploaded successfully. It's simpler: "What did the AI actually read?"

Notes

1. OpenAI, § “Vector stores”.↩

2. Google, § “Document understanding”.↩

3. Vladimir Karpukhin et al. (2020), pp. 6770–6771, §§2–3.1.↩

4. Lingnan Xu et al. (2025), p. 24608, Abstract & §1.↩

5. Lovisa Hagström et al. (2025), pp. 19694–19695, §§3.3–4.2.↩

6. Nelson F. Liu et al. (2024), p. 157, Abstract; p. 161, §2.3.↩

7. Taewhoo Lee et al. (2025), p. 5497, Abstract; pp. 5503–5504, §§5–5.1.↩

8. Yufeng Du et al. (2025), p. 23281, Abstract; pp. 23282–23283, §3.↩

9. Dongyu Ru et al. (2024), §§3.3.2–3.3.3, PDF pp. 4–5; §4.3, PDF pp. 7–8.↩

10. Cheng Niu et al. (2024), p. 10862, Abstract.↩

11. Tianyu Gao et al. (2023), p. 6468, §3.3; p. 6472, §6.↩

12. Nelson F. Liu et al. (2023), p. 7001, Abstract; p. 7007, §4.2.↩

13. Ian van Dort & Maria Heuss (2026), pp. 466–468, §§4.1–4.4; p. 470, §5.↩