The missing exception: a document retrieval experiment

The missing word was a permission boundary

A search result about exports can contain the right vocabulary and still omit the sentence a reader needs: “Guests cannot export a workspace.” In our synthetic retrieval lab, overlapping text windows retrieved a fragment mentioning exports, a workspace, and the beginning of the guest section. It did not contain the guest restriction. The terms were relevant; the evidence was incomplete.

We wrote one fictional policy document, split it three ways, and asked four fixed queries. The retrieval rule stayed the same: rank chunks by shared lowercase word tokens and break ties by original position. Non-overlapping eighty-character windows retained the required answer sentence in one of four top results. Overlapping windows retained it in three. Paragraph chunks retained it in four.

These are observed results from an executed local experiment. They are not a benchmark of embeddings, a commercial search product, or a language model. The narrow question is whether a simple retrieval step preserves a complete target sentence under these exact chunking choices. That boundary is useful to inspect before asking a model to explain a retrieved policy.

Original TrendsWhat chart from the downloadable results. The metric is exact target-sentence presence, not answer correctness or general retrieval quality.

What chunking changes

A retrieval workflow often divides a long document into pieces before searching it. Those pieces are commonly called chunks. The choice of boundary determines which facts travel together. If a rule and its exception land in different pieces, finding the rule may not retrieve the exception. If a heading is separated from its paragraph, a search result may lose the population to which the paragraph applies.

In a policy document, that loss can change meaning. Seven days for a trial workspace and thirty days for a paid workspace are different rules. A fragment containing only “exports are available” and a duration needs its scope. A system that combines fragments without preserving the distinction can produce an answer that sounds coherent but applies the wrong policy.

Our experiment isolates the preparation stage. There is no generated answer to distract from what the retrieval step delivered. We can inspect the chosen chunk and ask whether the needed sentence is present. If it is absent, better wording in the final answer cannot substitute for the missing source.

The fictional document

The document contains four paragraphs. Trial workspaces have seven-day export availability. Paid workspaces have thirty-day availability and an administrator control. Guests can read shared notes but cannot export a workspace. Deleting a workspace starts a fourteen-day recovery period, which is distinct from export availability.

Those policies were invented specifically for this lab. They do not describe TrendsWhat, a real service, or a provider's current terms. The numbers provide contrasting facts so that missing scope is easy to notice. Each paragraph contains related sentences, making paragraph boundaries an intentionally meaningful baseline.

The four queries ask about trial export duration, paid export duration, guest export access, and deletion recovery. Each query has a target sentence authored into the fixture. A result passes only if its selected chunk contains that sentence, ignoring case. This definition rewards complete evidence presence, not a model's ability to guess an answer from partial wording.

Three strategies, one search rule

The first strategy takes eighty characters at a time without overlap. The second also uses eighty-character windows, but consecutive windows overlap by thirty characters. The third uses the four original paragraphs. The first two strategies operate on characters, not model tokens; that distinction is important when interpreting the experiment.

For all strategies, the ranking code extracts lowercase alphabetic words into a set. It counts how many query words appear in each chunk. Repeated words do not add extra credit. When scores tie, the earlier chunk wins. This is intentionally simple lexical matching, with no stemming, synonym expansion, embeddings, or learned reranker.

Holding that rule constant lets us examine a specific interaction between boundaries and ranking. It does not make the comparison equal in every respect. Paragraphs are longer than the fixed windows, so they have more opportunities to contain query words and complete evidence. That length difference is a confound we retain and disclose rather than pretending the experiment isolates semantic boundaries perfectly.

What the local run found

Overlap rescued the paid and deletion targets in this setup. It did not rescue the guest target. The selected overlapping fragment combined the end of the paid paragraph with the beginning of the guest heading. It matched useful query terms but stopped before the restriction. The full selected text is preserved in the results, so readers can inspect the miss instead of trusting a label.

The paragraph strategy kept the guest heading, read permission, and export prohibition together. That is the most informative outcome for a workflow designer. A document structure carrying meaning also provided a useful retrieval boundary here. But four selected questions are much too few to support a general claim that paragraphs always outperform fixed windows.

Reproduce and inspect the miss

Download the script and run python3 run-labs.py --output results.json. Inspect the retrieval object. It includes the complete fictional document, every chunk for every strategy, all queries, target sentences, selected chunks, and scores. The recorded run also captures interpreter version and execution time.

Start with the guest case under the overlapping strategy. Read the selected chunk without looking at the target. Could you answer whether guests may export from that fragment alone? Then inspect the complete paragraph. The difference shows why retrieval evaluation needs examples of what was returned, not only a count of matched keywords.

Next, change the guest query or the window size and rerun. Treat that as a new configuration. Preserve the original result rather than overwriting your only record. If one change improves the guest case and breaks the trial case, you have learned about a tradeoff rather than found a universally better setting.

A workflow for diagnosing document answers

When a document assistant gives a wrong answer, inspect the chain in order. First confirm that the source document contains the required information. Then inspect text extraction. A PDF table can lose relationships before chunking begins. Next inspect chunk boundaries, retrieved candidates, and the final context supplied to the model. Only then evaluate the generated answer.

This order prevents an expensive loop of prompt changes aimed at the wrong stage. If the guest restriction never reaches the model, an instruction to cite sources does not ensure the restriction will appear. Conversely, if the full paragraph arrives and the answer still says guests can export, the retrieval step may be adequate and the generation step needs attention.

Keep the evidence visible in the user interface where practical. A short answer accompanied by the exact supporting paragraph lets a reader verify scope. A link to a whole fifty-page document is less helpful when the disputed issue is a missing exception. The right evidence unit should be small enough to inspect and large enough to preserve meaning.

Copy this retrieval review sheet

For a nontechnical team, these columns can be maintained in a spreadsheet. An engineer can later automate parts of collection. Do not automate away the stage labels: they tell you which component to change and help prevent a successful fix in one stage from hiding a failure in another.

Designing a stronger follow-up

A better comparison would use more documents, different writing styles, tables, long paragraphs, and questions that require evidence across sections. It would compare strategies under a controlled context budget rather than allowing arbitrary length differences. It might retrieve several chunks and assess whether the combined evidence preserves both rules and exceptions.

Include questions that cannot be answered from the document. A retrieval system always returning a relevant-looking passage can be dangerous when the source is silent. The expected outcome should sometimes be “not established by this document.” This lab lacks such cases, so its perfect paragraph score says nothing about refusal or uncertainty behavior.

If you introduce embeddings, record the actual model and indexing settings. If you introduce a language model to answer, separately score factual correctness, source support, and preservation of limitations. Keep those metrics distinct from target-sentence presence. A chunk can contain the answer while the model misreads it, and a model can guess correctly from a chunk that lacks evidence.

Why the modest result is still useful

This lab is small enough to understand completely. Every miss can be traced to a chosen fragment. The overlapping strategy improved the count while leaving a meaningful permission question unresolved. That combination is more useful than a single headline score: it tells us both what changed and which failure remains.

For your next document workflow, preserve headings and closely related rules where possible, inspect retrieved text before tuning the answer prompt, and build tests around exceptions. Choose chunk settings using your documents and evidence budget rather than copying a universal number from a tutorial.

The transferable AI skill is locating the failure boundary. “The assistant answered incorrectly” is a starting observation. “The top-ranked fragment omitted the guest restriction” is a diagnosis that can guide a concrete change. This experiment provides a small, reproducible example of moving from the first statement to the second.

Sources, materials, and limits

• Original document, queries, and executable comparison.
• Complete recorded results, including selected fragments.
• Python regular expression reference: background for the simple lexical tokenization used in the script.
• One fictional document, four queries, unequal chunk lengths, lexical ranking, no embeddings, no language model, and no general performance claim.