Deterministic retrieval, and the parts of RAG that refuse to sit still

I asked a resume search system for candidates with at least five years of experience and got back someone with three. The retrieval was working exactly as designed, which was the problem: a shortlist that silently includes the wrong people is also silently excluding the right ones, and the second failure is invisible from both ends. The candidate never learns they were dropped, and the person reading the list is relying on it being complete with no way to check that it is. Somebody missing a callback because of an index parameter is not a tuning issue, and unpicking it took me through pgvector's index internals into a question I had not thought to ask: which parts of a retrieval stack are allowed to be approximate at all.

Embeddings encode proximity, not predicate satisfaction

"5 years of experience" and "3 years of experience" are near neighbours in embedding space. They share almost all of their tokens, they occupy the same semantic region, and by every measure a bi-encoder is trained to optimise they are similar. Under a filter they are opposites. The embedding is doing its job; the job is simply not the one I was asking it to do.

The same failure recurs across a whole family of query terms. Negation gets flattened, because "no Python experience" sits close to "Python experience". Identifiers, part numbers and version strings get treated as approximately equal to their neighbours, so v2.14.1 retrieves v2.14.0 happily. Anything where the answer depends on an exact discrete match rather than a region of meaning is a bad fit for cosine distance.

The rule I now hold to without exception is that hard constraints belong in SQL predicates and soft preferences belong in ranking, and the two must never be mixed. Once a requirement is genuinely binary, expressing it as a similarity threshold is a category error dressed up as a tuning problem. No amount of raising the threshold makes "at least five years" true of the three-year candidate; it just discards good candidates alongside the bad one.

The post-filter trap

Having decided to put hard constraints in SQL, I hit the second failure, which is subtler and which pgvector documents plainly in its own README: "With approximate indexes, queries with filtering can return less results since filtering is applied AFTER the index is scanned."

Read that carefully, because it describes a silent failure. You ask for 10 results. The HNSW index returns its 40 nearest neighbours. Your WHERE clause eliminates 35 of them. You get 5 rows back, your application renders 5 cards, and nothing anywhere raises an error. The system did not tell you that it looked at a small, unrepresentative slice of the candidate set and that the answer you wanted was in the part it never examined.

pgvector's fix for this is iterative scan, and it is worth knowing the exact knobs.

-- retrieval_settings.sql
-- Index build. m defaults to 16, ef_construction to 64; both are build-time
-- and changing either means a rebuild.
CREATE INDEX ON resumes USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);
 
-- Full-text side, same table.
CREATE INDEX ON resumes USING gin (search_tsv);
 
-- Session settings. ef_search defaults to 40 and is tunable per session with
-- no rebuild, which makes it the only quality/latency dial worth exposing.
SET hnsw.ef_search = 100;
 
-- Without this, a restrictive WHERE clause silently truncates the result set.
-- strict_order preserves exact distance ordering; relaxed_order is faster and
-- may return results slightly out of order.
SET hnsw.iterative_scan = strict_order;
 
-- Bounds on how hard iterative scan is allowed to work before giving up.
SET hnsw.max_scan_tuples = 20000;    -- default 20000
SET hnsw.scan_mem_multiplier = 1;    -- default 1
 
-- IVFFlat, if you use it instead: lists = rows/1000 up to 1M rows, and
-- sqrt(rows) above that. probes defaults to 1, which is almost never enough.
-- CREATE INDEX ON resumes USING ivfflat (embedding vector_cosine_ops)
--   WITH (lists = 1000);
-- SET ivfflat.probes = 10;

The bounds matter as much as the switch. max_scan_tuples at its default of 20,000 means iterative scan is not a promise of completeness, it is a budget, and when the budget runs out you are back to a truncated result set with no error. What changes is that you now know the shape of the failure and can log when the budget was exhausted, which is the whole game.

The check I added around all of this is embarrassingly simple and has been worth more than the tuning. If the query asked for k results and fewer than k came back, the response carries a flag saying the result set may be truncated by filtering rather than exhausted by the corpus. Those are very different statements, and the raw row count cannot distinguish them. Surfacing the difference costs one boolean and turns a silent recall loss into something a caller can act on, which is the same move as treating an empty SQL result as suspicious rather than as an answer.

Why this is a structural bind rather than an implementation gap

I spent a while assuming pre-filtering was the obvious correct answer that pgvector had simply not implemented well. It isn't.

Pre-filtering gives you full recall over the filtered subset, but there is no query-specific index for an arbitrary predicate, so you end up brute-force scanning whatever the filter selected. Post-filtering is fast because the index does the work, but it over-fetches by an unknown factor and loses recall in a way that depends on how correlated your filter is with your vector neighbourhood. Neither side is a bug. The bind is real, and it comes from three properties of ANN structures: they optimise for proximity rather than boolean predicate satisfaction, deleting nodes fragments the HNSW graph, and you cannot jointly index a vector and its metadata the way a relational engine indexes a composite key.

There is active work on this. Curator (arXiv 2601.01291) and JAG (arXiv 2602.10258) both attack filtered ANN directly, and arXiv 2606.14193 makes the argument that existing filtered-ANN benchmarks are unrealistic enough that reported progress may not transfer. I read that last one as a caution against assuming the problem is about to be solved out from under you.

Fusing ranks instead of scores

The other half of the design is combining lexical and vector retrieval, where the standard mistake is trying to add a BM25 score to a cosine distance. Those two quantities share no scale, and the relationship between them is not stable across queries, so any weighted sum you write is a set of magic constants that will drift.

Reciprocal Rank Fusion sidesteps that. The score is a sum over rankers of 1 / (k + rank(d)), with k conventionally set to 60. The technique is usually attributed to Cormack, Clarke and Buettcher at SIGIR 2009, and I'm taking both that attribution and that constant from secondary sources rather than from the paper itself. Neither carries the argument, which stands on the mechanism: fusing ranks rather than scores means the only thing two retrievers need in common is an ordering, and orderings are always comparable.

-- hybrid_search.sql
-- One query, two retrievers, RRF combine. The metadata predicates are pushed
-- into BOTH CTEs: filtering only one side reintroduces the post-filter trap
-- through the back door.
WITH params AS (
  SELECT
    $1::vector      AS q_vec,
    $2::text        AS q_text,
    $3::int         AS min_years,
    $4::text[]      AS locations,
    60::float       AS k,
    200::int        AS depth
),
lexical AS (
  SELECT r.id,
         row_number() OVER (
           ORDER BY ts_rank_cd(r.search_tsv, websearch_to_tsquery('english', p.q_text)) DESC
         ) AS rank
  FROM resumes r, params p
  WHERE r.search_tsv @@ websearch_to_tsquery('english', p.q_text)
    AND r.years_exp >= p.min_years
    AND r.location = ANY (p.locations)
    AND r.work_auth = true
  LIMIT (SELECT depth FROM params)
),
semantic AS (
  SELECT r.id,
         row_number() OVER (ORDER BY r.embedding <=> p.q_vec) AS rank
  FROM resumes r, params p
  WHERE r.years_exp >= p.min_years
    AND r.location = ANY (p.locations)
    AND r.work_auth = true
  ORDER BY r.embedding <=> p.q_vec
  LIMIT (SELECT depth FROM params)
)
SELECT COALESCE(l.id, s.id) AS id,
       COALESCE(1.0 / (p.k + l.rank), 0.0)
     + COALESCE(1.0 / (p.k + s.rank), 0.0) AS rrf_score,
       l.rank AS lexical_rank,
       s.rank AS semantic_rank
FROM lexical l
FULL OUTER JOIN semantic s ON s.id = l.id
CROSS JOIN params p
ORDER BY rrf_score DESC
LIMIT 20;

Two honest notes about that query. ts_rank_cd is cover density ranking, which is a reasonable lexical signal but is not BM25, and if you want real BM25 inside Postgres then ParadeDB's pg_search is the thing that provides it. And returning both component ranks alongside the fused score is not decoration: when a result looks wrong, the first question is always whether it came from the lexical side, the semantic side, or scraped in from both, and you cannot answer that after the fact if you threw the ranks away.

For the shape of the result, BEIR (NeurIPS 2021 Datasets and Benchmarks) is the citable source: dense retrievers that win in-domain lose out-of-domain, and BM25 remains a highly competitive zero-shot baseline across its 18 datasets. I'm deliberately not attaching a decimal to that, in either direction. The figures people quote for how far dense beats BM25, or what hybrid adds on top, are aggregates I cannot trace back to a paper, and the ones for 2026 hybrid stacks live in blog leaderboards rather than anywhere I can check. Shape, not measurement.

Determinism is a stack property

Here is where my mental model was most wrong. I had assumed retrieval was the deterministic half of the system and that any run-to-run variation had to be coming from the model.

pgvector says otherwise, in one sentence in its README: "Unlike typical indexes, you will see different results for queries after adding an approximate index." That is the entire warning, and it is enough. The moment you build an approximate index, retrieval stops being a function of the query alone and becomes a function of the index's internal state, and the documented behaviour is that identical queries can return different rows. Anything downstream that changes between runs may be changing because of that, with the model behaving identically.

The generation side turns out not to be a fixed point either, and here I'm reporting rather than verifying. Thinking Machines Lab's "Defeating Nondeterminism in LLM Inference" reports that sampling 1,000 completions from Qwen3-235B-Instruct at temperature 0 produced 80 distinct outputs, with the first divergence at token 103, and that batch-invariant kernels restore bitwise-identical output across 1,000 runs on Qwen3-8B at roughly a 61.5% throughput cost, falling to about 34.35% once CUDA graphs are applied. I have not reproduced any of those figures.

The mechanism they describe is the part I'd act on, because it can be reasoned about without trusting the numbers. Matmul, RMSNorm and attention kernels change their floating-point accumulation order depending on batch size, batch size depends on how many concurrent requests the server is handling, and floating-point addition is not associative. Your output therefore depends on other people's traffic.

Seeds do not fix that. A seed pins the sampler, and the divergence sits upstream of sampling in the kernels themselves.

What I do about it is not to chase bitwise determinism end to end, which is expensive and mostly unnecessary. It is to make retrieval deterministic and let only generation vary, so that when an answer changes I know which layer changed. Concretely: pin the embedding model version and store it per row, because re-embedding a corpus with a new model version silently invalidates every stored vector and absolutely nothing errors; pin and log ef_search per query class so a latency tuning change shows up in the audit trail rather than as mysterious quality drift; use RRF, whose rank-based fusion means small score perturbations don't reorder results; and snapshot the corpus by version so an eval run is reproducible against the corpus it was scored on.

Resume search, concretely

The pipeline that came out of all this parses each resume, extracts typed columns, and embeds the free text separately, so that the structured and unstructured parts of a document are queried by the mechanisms suited to each.

# query_plan.py
from dataclasses import dataclass, field
from typing import Literal
 
Outcome = Literal["answer", "clarification_request", "refusal"]
 
 
@dataclass
class HardConstraints:
    """Every field here becomes a SQL predicate. Nothing here is negotiable,
    and nothing here is ever expressed as a similarity threshold."""
    min_years_exp: int | None = None
    work_authorized: bool | None = None
    locations: list[str] = field(default_factory=list)
    required_skills: list[str] = field(default_factory=list)  # join table, exact
 
 
@dataclass
class SoftPreferences:
    """Ranking signals only. These can move a candidate up or down; they can
    never remove one from the result set."""
    domain_text: str = ""          # "fintech backend, high write throughput"
    prefers_skills: list[str] = field(default_factory=list)
    seniority_hint: str | None = None
 
 
@dataclass
class Plan:
    kind: Outcome
    hard: HardConstraints | None = None
    soft: SoftPreferences | None = None
    question: str | None = None    # set when kind == clarification_request
    reason: str | None = None      # set when kind == refusal
 
 
def plan_from_request(req: str) -> Plan:
    """Extraction step. If a stated requirement cannot be mapped onto a typed
    column, it does not silently become a ranking signal: it becomes a
    clarification request, because demoting a hard constraint to a soft one is
    how you end up returning a three-year candidate for a five-year search."""
    ...

That last docstring is the design in one sentence. The three-armed output type applies to the whole system, and the clarification arm has to fire often enough that it is a real path rather than an unreachable branch. If a user asks for "senior engineers near the office" and neither seniority nor proximity is a typed column, the honest response is a question, not a ranked list that quietly reinterprets the request.

Two published findings point the same way, and I have both secondhand rather than from the papers. arXiv 2602.18550 is reported to find that many models cannot consistently select the resume describing the more qualified candidate, and do not reliably abstain when candidates are equally qualified. That second half is the more damning of the two, because a system that expresses a preference between two equivalent candidates is generating a preference rather than detecting one. Separately, Wilson and Caliskan, published at AIES, are reported to find that LLM resume screening disadvantages Black-associated and female-associated names with all other content held identical.

I'd have made the same design call without either result, because a ranker whose reasoning you cannot inspect is already a problem before you know what it is keying on. What the findings add is a specific account of what it might be keying on, and that makes this a correctness argument as much as an ethical one. A name carries no information about the question being asked, so a ranker that responds to one is responding to noise. Hard filters over typed columns plus an auditable ranking function is a design where you can inspect what moved a candidate up or down. An end-to-end model judgement is one where you cannot, and "we could not tell you why" is not an answer that survives being asked about a specific person.

Measuring retrieval on its own terms

The metric that dominates everything downstream is recall@k, because the generator cannot cite what it never saw. Whatever your answer quality is, it is bounded above by whether the right document made it into the context window. MRR is the right choice when there is exactly one correct answer and its position matters, and nDCG@k is the right one for graded relevance, which is the actual shape of resume ranking where candidates are better and worse rather than right and wrong.

The apparatus that made this tractable was small: a golden set of somewhere between 50 and 200 query-to-expected-document pairs drawn from the actual corpus rather than a public benchmark, run in CI on every pull request that touches retrieval, with one variable changed at a time. Public benchmarks tell you about the general shape of retrieval behaviour. They tell you nothing about whether your chunking change broke your corpus.

Building that set is the tedious part and there is no way around it. I labelled mine by hand, one query at a time, writing down which documents genuinely should have surfaced before looking at what the system returned, because doing it in the other order produces a golden set that ratifies the behaviour you already have. A hundred honestly labelled pairs from your own corpus beat any quantity of borrowed relevance judgements, and the labelling surfaces query ambiguities no metric would show you.

One trap worth naming: a change that improves nDCG while dropping recall@50 is a regression, and an aggregate score will hide it. You have made the top of the list prettier while removing documents from consideration entirely, which is exactly the trade you don't want in a system whose job is to find candidates. I gate on recall separately for that reason, and treat a recall drop as a blocking failure even when the headline number went up.

What actually holds still

Retrieval determinism is achievable and worth paying for. Generation determinism depends on kernel-level batch invariance and is mostly not worth chasing. The useful configuration is the one where retrieval is pinned, logged and reproducible, so that when an answer changes you can say the corpus, the index parameters and the fused ranking were all unchanged, and therefore the model moved. That is a debuggable system. The alternative, where four layers vary at once, is a system where every investigation starts from zero.