---
title: "When the documents disagree"
description: "An answer cited a specification section by number and was wrong, because an addendum had already revised that clause and the vector index had no model of temporal precedence. A technical account of hierarchical document routing, supersession graphs, and NLI verification."
date: "March 17, 2026"
url: "https://himanshuat.com/blogs/when-the-documents-disagree"
---
# When the documents disagree

The answer appeared completely verified: it cited a technical specification section by exact number, quoted the clause, and was factually wrong.

An addendum issued weeks earlier had revised that requirement. The vector index treated all chunks as a flat semantic pool with no concept of temporal supersession, so the retriever surfaced the stale clause with high cosine similarity.

In engineering or finance workflows, acting on a superseded requirement means rework, and often liability.

> A citation is not evidence; it is a claim about evidence. An index with no model of supersession will produce a true citation to a clause that no longer governs.

### Chapter 0: The three structural dimensions of an index

A retrieval index is defined by three structural decisions:

1. **Retrieval unit:** A fixed-size character chunk, a semantic sentence group, a document section, or a structured page range.
2. **Search scope:** A single monolithic vector store spanning the entire corporate corpus versus isolated, partitioned stores per document.
3. **Relational metadata:** Explicit relational graphs linking chunks (hierarchies, revisions, supersessions) versus an isolated bag of vectors.

---

### Step 1: Establish partitioned index boundaries

On complex corpora, the physical boundary of the vector index has a larger impact on answer accuracy than embedding model swaps.

In [FinanceBench](https://arxiv.org/abs/2311.11944) — Islam, Kannappan, Kiela, Qian, Scherrer and Vidgen, 2023 — 10,231 financial questions were written against 361 public filings from 2015 to 2023, and 150 of them were manually reviewed and run across model configurations. Changing only the retrieval index boundary moved the results this far:

| Retrieval Condition | Correct | Incorrect | Refusal / No Answer |
|---|---|---|---|
| Shared vector store over entire corpus | 29 (19%) | 20 (13%) | 101 (68%) |
| Partitioned store per document | 75 (50%) | 17 (11%) | 58 (39%) |
| Whole document in LLM context | 118 (79%) | 26 (17%) | Not reported |
| Oracle (exact ground-truth evidence pages supplied) | 128 (85% ceiling) | 22 (15%) | Not reported |

Partitioning vectors per document increased retrieval accuracy from 19% to 50% on the exact same questions with zero modifications to the model weights.

**The diagnostic check:** if you know your embedding dimensionality and reranker model but cannot state how your index partitions documents, you are optimizing the wrong layer.

---

### Step 2: Establish the oracle accuracy ceiling

Two critical data points emerge from the FinanceBench evaluation:

1. **The generator accuracy ceiling:** When GPT-4-Turbo is supplied with the exact ground-truth evidence pages (the oracle condition), it still fails on 22 of 150 questions (85% ceiling). Retrieval architectures cannot lift end-to-end accuracy above the accuracy of the generator given perfect context.
2. **Refusal conversion in long-context models:** Passing entire documents directly to large context windows (128k+) achieved 79% accuracy, but incorrect answers rose from 13% to 17%. Long-context injection converted explicit refusals into confident hallucinations.

> An explicit refusal is a low-cost failure. A confident hallucination is an expensive failure. A system without an explicit refusal path defaults to high-cost failures.

---

### Step 3: Traverse author-defined document hierarchies

Instead of computing bottom-up clusters over flat text chunks, structured documents should be indexed using their inherent table-of-contents hierarchy (as in [PageIndex](https://github.com/VectifyAI/PageIndex)-style tree traversal, which replaces the vector index with a hierarchical tree and has an LLM reason its way down it).

```python
# node.py
from dataclasses import dataclass

@dataclass
class Node:
    """A hierarchical document node preserving exact page bounds for verifiable citations."""
    title: str          # e.g., "03 30 00 Cast-In-Place Concrete"
    node_id: str        # Stable identifier
    start_index: int    # First page
    end_index: int      # Last page
    summary: str        # Structural summary for navigation
    children: list["Node"]

def descend(root: Node, question: str, budget: int = 4) -> list[Node]:
    """Walks the hierarchy using top-down semantic routing over node summaries."""
    selected, frontier, depth = [], [root], 0
    while frontier and depth < budget:
        menu = [
            {"node_id": n.node_id, "title": n.title, "summary": n.summary}
            for node in frontier for n in node.children
        ]
        if not menu:
            break
        chosen_ids = ask_model_to_select(question, menu)
        frontier = [n for n in all_nodes(root) if n.node_id in chosen_ids]
        selected.extend(frontier)
        depth += 1
    return selected

def cite(node: Node) -> str:
    """Generates structured citation directly from traversed node metadata."""
    return f"{node.title}, pp. {node.start_index}-{node.end_index}"
```

In technical corpora governed by industry standards (such as the Construction Specifications Institute's [MasterFormat](https://www.csiresources.org/standards/masterformat), which organizes project documentation into numbered work-result sections), navigating structured headings directly maintains inspectability: if retrieval selects the wrong section, you can review the exact branch summary that misrouted the search.

---

### Step 4: Model supersession as an explicit relational graph

When revisions, addenda, or change orders enter a corpus, temporal precedence must be modeled as a relational graph rather than a cosine similarity score.

```sql
-- index_schema.sql
CREATE TABLE spec_node (
  node_id       text PRIMARY KEY,
  project_id    text NOT NULL,
  parent_id     text REFERENCES spec_node(node_id),
  csi_number    text,                -- '03 11 00'
  division      int GENERATED ALWAYS AS (substring(csi_number,1,2)::int) STORED,
  section_part  text,                -- 'Part 2 Products'
  title         text NOT NULL,
  page_start    int  NOT NULL,
  page_end      int  NOT NULL,
  source_doc_id text NOT NULL REFERENCES source_doc(doc_id),
  summary       text
);

CREATE TABLE source_doc (
  doc_id     text PRIMARY KEY,
  project_id text NOT NULL,
  doc_type   text NOT NULL,          -- spec, drawing, addendum, change_order
  issued_on  date NOT NULL,          -- Latest date governs within equivalent precedence
  sheet_no   text
);

-- Explicit supersession relationship
CREATE TABLE supersedes (
  superseding_node text REFERENCES spec_node(node_id),
  superseded_node  text REFERENCES spec_node(node_id),
  effective_on     date NOT NULL,
  scope            text NOT NULL,    -- 'full' | 'partial'
  PRIMARY KEY (superseding_node, superseded_node)
);

CREATE INDEX ON spec_node (project_id, csi_number);
CREATE INDEX ON supersedes (superseded_node);
```

During query execution, candidate chunks from `spec_node` are checked against `supersedes`. If a node has an active superseding entry, the retrieval engine routes to the replacement clause automatically.

---

### Step 5: Verify claim entailment with Natural Language Inference

To prevent hallucinated citations, run post-generation verification using Natural Language Inference (NLI) models across exact character spans:

```python
# verify.py
from dataclasses import dataclass

@dataclass
class Span:
    doc_id: str
    page: int
    char_start: int
    char_end: int
    text: str

@dataclass
class Claim:
    text: str
    cites: list[Span]

def verify_claim_entailment(claim: Claim, nli_evaluator) -> dict:
    """
    Computes citation precision and recall using NLI entailment scoring.
    """
    per_cite = [nli_evaluator(premise=s.text, hypothesis=claim.text) for s in claim.cites]
    joint = nli_evaluator(
        premise="\n".join(s.text for s in claim.cites),
        hypothesis=claim.text,
    )
    supporting = sum(1 for r in per_cite if r == "entailment")
    contradictions = [s for s, r in zip(claim.cites, per_cite) if r == "contradiction"]

    return {
        "citation_precision": supporting / max(len(claim.cites), 1),
        "citation_recall": 1.0 if joint == "entailment" else 0.0,
        "contradicted_by": contradictions,
    }
```

```mermaid
flowchart TD
  Q[question] --> R[hierarchical descent]
  R --> V{NLI entails claim?}
  V -->|no| REF[explicit refusal]
  V -->|contradiction| RFI[escalate to conflict workflow]
  V -->|yes| S{single governing document?}
  S -->|no| RFI
  S -->|yes| A[generate verified answer with exact citations]
```

When an irreconcilable conflict between two active documents is detected, the pipeline must escalate the discrepancy to a human operator or formal Request for Information (RFI) rather than silently picking an arbitrary winner.

---

### Key production failure modes

- **Parser table extraction loss:** If the OCR or layout parser omits cells during schedule extraction, downstream retrieval fails regardless of vector indexing quality.
- **Parametric knowledge overriding context:** Large models frequently default to common industry averages (such as standard steel yield strength) when domain documents specify non-standard specifications. Enforce strict temperature and system prompt grounding.
- **Missing character offsets:** Retain exact byte and character offsets from the raw PDF parser throughout chunking. Losing character coordinates makes downstream automated verification unreliable.

---

### When this is the wrong choice

- **Nothing in the corpus gets revised.** The `supersedes` table and the precedence check exist because addenda and change orders arrive after the spec does. On a corpus that is written once and never amended, you are maintaining a join that no query needs, and the index boundary work in Step 1 gets you most of the accuracy anyway.
- **The documents have no author-defined hierarchy.** The descent in Step 3 routes over table-of-contents nodes and their summaries. Chat transcripts, email threads, and scanned field notes have no such tree, so there is nothing to traverse and no branch summary to inspect when retrieval goes wrong. Flat chunking is the honest choice there.
- **The governing document fits in context.** Passing whole documents to a long-context model reached 79% on FinanceBench against 50% for the partitioned retriever, and it is far less to build. Read the other half of that number before you take the trade: incorrect answers went from 13% to 17%, because the refusals became confident answers instead.
- **A wrong answer is cheap and a refusal is expensive.** NLI verification costs a model call per citation plus a joint call, and its output is a refusal path. That is the right trade when acting on the answer is expensive. In exploratory search, where a person reads the source anyway, it is latency spent making the system say less.

---

Source: https://himanshuat.com/blogs/when-the-documents-disagree