When the documents disagree

The answer was correct, cited a specification section by number, and was wrong. An addendum issued weeks earlier had revised that clause, and nothing in the index knew that one document supersedes part of another, so the citation attached to it made the answer look checked. Someone reads an answer like that and orders material against it, and the cost of being wrong is measured in rework and, if it runs far enough, in who pays for it. What I needed to understand was how a system with correct retrieval mechanics and a real citation could still be this wrong.

The retrieval architecture was the bug

The result that explained it has nothing to do with construction. The FinanceBench paper by Islam et al., from Patronus AI, Contextual AI and Stanford, works over 10,231 questions across 361 filings from 40 companies, with a 150-question human-evaluated sample. On that sample, GPT-4-Turbo querying a single shared vector store over the corpus, built with Chroma through LangChain on ada-002 embeddings, got 29 answers correct (19%), 20 incorrect (13%), and failed to answer 101 (68%).

Give each document its own store and the same model on the same questions gets 75 correct (50%), 17 incorrect (11%), 58 failed (39%). Nothing about the model changed. The failure was retrieval architecture, and I keep returning to that pair of numbers because of how much attention goes to model selection and prompt phrasing, both of which were held constant across a result that more than doubled accuracy.

The ceiling that nobody quotes

Two further numbers deserve as much attention as the headline.

The first is the oracle condition. Hand GPT-4-Turbo the exact evidence pages, removing retrieval from the problem entirely, and it still gets 22 of 150 wrong, a 15% error rate. The ceiling is 85%, not 100%. Any claimed accuracy above that on this benchmark is describing a system where something other than retrieval also improved, and that is a load-bearing observation I'll come back to.

The second concerns long context. Feeding the whole document in gives 118 correct (79%) with 26 incorrect (17%). Compare that to the shared-store configuration's 13% incorrect and you find that long context did not remove errors so much as convert refusals into wrong answers. The failures stopped announcing themselves. Prompt order matters enormously in that setting too: 78% versus 25% for GPT-4-Turbo depending on whether the context precedes or follows the question, a swing large enough that any long-context comparison not controlling for it is measuring the wrong thing.

The authors put the asymmetry plainly: "Models refusing to answer is arguably preferable to giving an incorrect answer as it creates less risk of error, and misplaced trust, by users." That sentence is the design principle for everything that follows. The output type I want from a document QA system is answer | clarification_request | refusal, and the refusal arm has to fire often enough to be a real path rather than a branch that exists only in the type signature.

PageIndex, read carefully

The most interesting alternative I've worked with is PageIndex, which deserves a careful reading rather than a credulous one.

Its README describes two steps: "(1) Generate a 'Table-of-Contents' tree structure index of documents; (2) Perform (agentic) reasoning-based retrieval through tree search." There is no vector store anywhere in that description. Ingestion runs an LLM pass over the document to build a tree mirroring its table of contents; retrieval hands the model that tree of titles and summaries, lets it select node IDs, and fetches only those page ranges.

The node schema is where the value sits.

# node.py
from dataclasses import dataclass
 
 
@dataclass
class Node:
    """A PageIndex-style node. The two index fields are page numbers, which is
    the property that makes every retrieval result citable by construction."""
    title: str          # "03 30 00 Cast-In-Place Concrete"
    node_id: str        # stable id the model selects during tree descent
    start_index: int    # first page
    end_index: int      # last page
    summary: str        # what the model reads while navigating
    children: list["Node"]
 
 
def descend(root: Node, question: str, budget: int = 4) -> list[Node]:
    """Walk the tree by reasoning rather than by distance. Each level costs one
    sequential LLM call, and the calls cannot be parallelised because the next
    level depends on the selection made at this one."""
    selected, frontier, depth = [], [root], 0
    while frontier and depth < budget:
        menu = [
            {"node_id": n.node_id, "title": n.title, "summary": n.summary}
            for node in frontier for n in node.children
        ]
        if not menu:
            break
        chosen_ids = ask_model_to_select(question, menu)   # returns node_ids
        frontier = [n for n in all_nodes(root) if n.node_id in chosen_ids]
        selected.extend(frontier)
        depth += 1
    return selected
 
 
def cite(node: Node) -> str:
    """No post-hoc citation extraction is needed: the title and page range came
    back with the content, because they are what the retrieval selected on."""
    return f"{node.title}, pp. {node.start_index}-{node.end_index}"

Every result carries a section title and a page range because those are the fields retrieval navigated by. The usual approach retrieves a chunk and then reverse-engineers which page and heading it came from, which is a lossy reconstruction of information chunking already discarded. Citation here is structural rather than bolted on, and that matters more where a citation is a contractual reference than it does in general question answering.

The claim, and its problems

The number attached to PageIndex is that "Mafin 2.5, powered by PageIndex, achieved a state-of-the-art 98.7% accuracy on FinanceBench." I want to be exact about what is knowable there.

The evaluated subset is not disclosed. Neither is the question count, the judge, nor the generator model. FinanceBench's full open-source set and its 150-question human-evaluated sample are different things, and a figure quoted without saying which it refers to cannot be compared against the paper's numbers.

The harder problem is arithmetic. FinanceBench's oracle ceiling is 85%: the model with perfect retrieval still got 15% wrong. A system reporting 98.7% is therefore reporting something that cannot be attributed to retrieval alone, because retrieval cannot lift a system above the accuracy its generator achieves when handed the right pages. Either the generator is substantially better than the GPT-4-Turbo the paper evaluated, or the evaluation subset differs, or the judging criteria differ. Probably some of each. The claim may well be honest; it does not support the inference that retrieval architecture produced it.

There is a third scoping issue that the README itself states. The open-source package uses standard PDF parsing, while enhanced OCR and tree building are offered as a cloud service. The open-source package and the thing that scored 98.7% are not the same artifact, and anyone benchmarking the former against that number is comparing two different systems. It is also worth noting where the figure does not appear: pageindex.ai itself carries no benchmark claims, so this number circulates in write-ups rather than sitting on the project's own site as something it stands behind.

On the operational side there is nothing to cite, because no latency, throughput or cost figures are published anywhere I can find. What I can say comes from the structure rather than from a source. Every query costs multiple sequential LLM calls to walk the tree, descent cannot be parallelised because each level depends on the selection made at the level above, and ingestion requires an LLM pass over the whole document. Those are properties of the design, not measurements I've taken. The remaining limitation I've seen repeated, that scaling past a few hundred documents is unproven, I'm passing on secondhand and have not tested at that size.

Relocating the approximation rather than removing it

The framing I find fair, and which I think is the actual intellectual content here, is this. Vector search approximates relevance through embedding geometry. PageIndex approximates relevance through LLM reasoning over summaries. Both can be wrong, and replacing one with the other does not eliminate approximation, it moves it.

What changes is inspectability. When a vector search returns the wrong chunk, the explanation is a distance in a space you cannot read. When a tree walk returns the wrong section, you can read the path it took, see which summary misled it, and fix that summary. That is a genuine advantage, and a smaller claim than the one usually made on this technology's behalf.

Induced hierarchies versus the author's own

The comparison most write-ups skip is RAPTOR (arXiv 2401.18059, ICLR 2024), which also builds a tree over documents. What follows is my reading and not an established result: neither project makes this comparison, and I am describing RAPTOR's mechanism from secondary reading rather than from a careful pass over the paper.

RAPTOR is described as working bottom-up: embed the chunks, cluster them with a Gaussian mixture model, summarise each cluster, repeat over the summaries, then retrieve by embedding similarity across all levels of the resulting tree. PageIndex works top-down, taking the document's own table of contents as the structure and navigating it by reasoning. The direction of construction is the whole difference.

The distinction that matters is that RAPTOR's hierarchy is induced. Its clusters are statistical artifacts, and no node in that tree corresponds to anything a human would cite. PageIndex's hierarchy is the author's own, which is exactly why its nodes are citable. For a specification book where CSI section numbering is the ground-truth hierarchy, running a clustering algorithm over the text throws away the answer key that came printed in the document.

A spec book already knows its own structure

As it is commonly described, and I have this from summaries rather than from the standard itself, CSI MasterFormat organises construction specifications into 50 divisions, expanded from 16 in November 2004, with six-digit numbering in a Division-LevelTwo-LevelThree pattern. Section 03 11 00 is Division 03 Concrete, subgroup 03 11 Concrete Forming and Accessories. Inside a section, SectionFormat imposes a fixed three-part structure: Part 1 General, Part 2 Products, Part 3 Execution.

That structure already exists, is authored deliberately, and is what a human uses to navigate. Inferring one over the top of it is work spent discarding information.

-- index_schema.sql
-- The document tree, with supersession as a first-class relation rather than
-- something the ranker is expected to figure out.
CREATE TABLE spec_node (
  node_id       text PRIMARY KEY,
  project_id    text NOT NULL,
  parent_id     text REFERENCES spec_node(node_id),
  csi_number    text,                -- '03 11 00'
  division      int GENERATED ALWAYS AS (substring(csi_number,1,2)::int) STORED,
  section_part  text,                -- 'Part 2 Products'
  title         text NOT NULL,
  page_start    int  NOT NULL,
  page_end      int  NOT NULL,
  source_doc_id text NOT NULL REFERENCES source_doc(doc_id),
  summary       text
);
 
CREATE TABLE source_doc (
  doc_id     text PRIMARY KEY,
  project_id text NOT NULL,
  doc_type   text NOT NULL,          -- spec | drawing | addendum | change_order
                                     -- | agreement | schedule
  issued_on  date NOT NULL,          -- among same-level docs, latest controls
  sheet_no   text                    -- drawings only
);
 
-- The relation that has to be explicit. An addendum does not simply mention a
-- section; it replaces part or all of it as of its issue date.
CREATE TABLE supersedes (
  superseding_node text REFERENCES spec_node(node_id),
  superseded_node  text REFERENCES spec_node(node_id),
  effective_on     date NOT NULL,
  scope            text NOT NULL,    -- 'full' | 'partial'
  PRIMARY KEY (superseding_node, superseded_node)
);
 
CREATE INDEX ON spec_node (project_id, csi_number);
CREATE INDEX ON supersedes (superseded_node);

A hybrid search that surfaces a superseded clause and its addendum-revised replacement with equal confidence and no temporal ordering is worse than returning nothing, because it produces a defensible-looking answer nobody will re-check. Supersession has to be a relation in the index, enforced at query time, rather than a signal the ranker is trusted to pick up from wording.

Precedence is a contract term, not a heuristic

Construction contracts usually specify an order of precedence among documents. A representative ordering puts change orders and written amendments first, then the Agreement, then drawings, where large scale governs over small scale, then specifications and addenda issued before execution, then owner-furnished information, then other listed Contract Documents. Among documents at the same level, the latest dated one controls. Federal contracts have their own rule in FAR 52.236-21: "In the case of difference between drawings and specifications, the specifications shall govern."

The nuance that stops this being a clean algorithm is that the AIA generally advises against precedence clauses at all, on the grounds that they remove the architect's interpretive autonomy. So whether a given project even has an order of precedence is project-specific, and a system that hardcodes one is asserting a contract term that may not exist in that contract.

Every specific in those two paragraphs is secondhand, taken from summaries rather than from contract documents or standard forms. That flag is the argument, not a hedge: if I cannot establish from secondary reading which precedence rule governs a particular project, a retrieval system reading the same material certainly cannot, and it has no business behaving as though it can.

Four modalities, one fact

Contradiction detection in the literature is framed as natural language inference, with its three-way entailment, neutral and contradiction judgement. The dataset lineage runs from SNLI, MNLI and ANLI, too shallow for this, through ContractNLI at the clause level, to ContraDoc and ECON at document scale. arXiv 2504.00180 was the most directly applicable, because it separates self-contradictions within one document from pairwise contradictions across documents, and those need different handling.

The taxonomy worth carrying into design is the split between inter-document conflicts, where two retrieved passages contradict each other, and parametric-contextual conflicts, where retrieved evidence contradicts what the model learned during training. The second is the quieter danger in a technical domain. A model that has absorbed a typical concrete compressive strength will smooth over a spec calling for something unusual, and produce an answer that is right about concrete in general and wrong about this building.

Construction makes it harder than the literature's framing. The same requirement appears as prose in the specification, as a dimension on a drawing, as a row in a schedule, and as a revision in an addendum. Four modalities, one fact, and an NLI model trained on sentence pairs has no purchase on the drawing or the schedule. Detecting that a dimension callout disagrees with a spec paragraph is not a text entailment problem in any form the benchmarks cover.

Which puts weight on parsing, and parsing is the unmeasured ceiling in most of these systems. OmniDocBench (CVPR 2025) is the resource I'd point at, described secondhand as 981 PDF pages across 9 document types, with a three-level evaluation protocol covering end-to-end performance, single-module performance for OCR, layout detection, table recognition, formula recognition and reading order, and attribute-based robustness. Its exact scale is not the point I need it for. The point is mine and holds without it: a RAG system's accuracy ceiling is its parser's accuracy, and almost nobody measures the parser separately. If table recognition drops a row from a door schedule, no amount of retrieval quality recovers it, and the failure will be attributed to the model.

Citations have to be checked, not simply emitted

ALCE (arXiv 2305.14627) is where the two metrics I now consider mandatory come from, as they are commonly described rather than as I read them in the paper. Citation recall asks whether every statement is fully supported by its cited passages. Citation precision asks whether each individual citation actually supports the claim it is attached to. Both are reported to be computed with an NLI model and validated against human judgment, and the definitions are what I use regardless of how the original evaluation was run.

# verify.py
from dataclasses import dataclass
 
 
@dataclass
class Span:
    """Raw offsets kept from the parser all the way through. If chunking loses
    these, verification becomes a fuzzy string match against the source."""
    doc_id: str
    page: int
    char_start: int
    char_end: int
    text: str
 
 
@dataclass
class Claim:
    text: str
    cites: list[Span]
 
 
def verify(claim: Claim, nli) -> dict:
    """Precision: does each cited span entail the claim on its own or jointly?
    Recall: is the claim fully supported by the union of its cited spans?"""
    per_cite = [nli(premise=s.text, hypothesis=claim.text) for s in claim.cites]
    joint = nli(
        premise="\n".join(s.text for s in claim.cites),
        hypothesis=claim.text,
    )
    supporting = sum(1 for r in per_cite if r == "entailment")
    return {
        "citation_precision": supporting / max(len(claim.cites), 1),
        "citation_recall": 1.0 if joint == "entailment" else 0.0,
        "contradicted_by": [
            s for s, r in zip(claim.cites, per_cite) if r == "contradiction"
        ],
    }

Keeping raw span offsets from the parser end to end is what makes that check possible, and it is what most chunking pipelines quietly destroy. A citation that exists but does not entail the claim is worse than no citation, because it manufactures verifiability: it invites trust in a link nobody will follow, and it survives every review that consists of checking whether citations are present.

Who is allowed to decide

All of which leads to the constraint that shapes the whole system. An agent that resolves a conflict between a specification and a drawing on its own is doing something with contractual and financial consequences, potentially change-order-sized. The correct behaviour when it detects a conflict is to surface it, cite both sources with section number and sheet number, state the applicable precedence clause if the contract contains one, and route the question to an RFI.

The contractor's obligation on discovering a conflict between contract documents is to report it to the architect through an RFI, not to pick the interpretation that seems more sensible. A system that silently picks is not being helpful. It is quietly assuming liability the contract assigns elsewhere.