RAG Foundations: From First Query to a Retriever You Can Actually Trust

The first RAG system I shipped took about a weekend. The retriever returned plausible-looking chunks, the generator produced fluent answers, and manual spot-checks looked fine.

Then I ran it on 200 representative queries with ground-truth answers. Hit Rate at k=5 was 51%. Nearly half the queries never retrieved the document that contained the answer.

The generator had been producing confident, well-written hallucinations.

Fluent output is not evidence of correct retrieval. Measure retrieval separately before you evaluate generation quality.

This post covers the decisions that actually moved retrieval accuracy: chunk boundary strategy, embedding model selection against real constraints, and the two metrics that replace guesswork.

Chapter 0: What retrieval failure looks like

Standard RAG is sequential: embed the query, fetch top-k chunks, construct a prompt, generate. The assumption is that top-k retrieval achieved adequate recall.

When it doesn't, the failure is invisible. The model synthesizes a paragraph using whatever was in the context window. If the relevant passage wasn't retrieved, the answer either hallucinates or hedges politely — both of which look reasonable to a reviewer who doesn't know the ground truth.

The earliest diagnostic: before touching chunk size or embedding models, run 50–100 representative queries against your index and check whether the correct document appeared in the results at all. If Hit Rate at k=5 is below 70%, fix retrieval before everything else.

Step 1: Fixed-size chunking and where it breaks

Fixed-size chunking splits text every N tokens, with an optional overlap window:

interface TextChunk {
  content: string;
  startIndex: number;
  endIndex: number;
  metadata: {
    sourceId: string;
    chunkIndex: number;
  };
}
 
interface ChunkingOptions {
  chunkSize: number;
  overlapSize: number;
}
 
function fixedSizeChunker(text: string, options: ChunkingOptions): TextChunk[] {
  const { chunkSize, overlapSize } = options;
  const chunks: TextChunk[] = [];
 
  if (chunkSize <= 0 || overlapSize < 0 || overlapSize >= chunkSize) {
    throw new Error(`Invalid options: chunkSize=${chunkSize}, overlapSize=${overlapSize}`);
  }
 
  let currentPos = 0;
  let chunkIndex = 0;
 
  while (currentPos < text.length) {
    const endPos = Math.min(currentPos + chunkSize, text.length);
    const chunkContent = text.slice(currentPos, endPos);
 
    if (chunkContent.trim().length > 0) {
      chunks.push({
        content: chunkContent,
        startIndex: currentPos,
        endIndex: endPos,
        metadata: { sourceId: 'doc-001', chunkIndex: chunkIndex++ },
      });
    }
 
    if (endPos === text.length) break;
    currentPos = endPos - overlapSize;
  }
 
  return chunks;
}

Fixed-size chunking works well on uniform prose. It fails on technical documents where key facts sit at section boundaries — the 512-token window cuts a sentence in half, half lands in one chunk and half in the next, and neither chunk contains a complete retrievable fact.

The failure mode to watch: overlap progress stalls. When a natural break snaps endPos back close to startPos, the next currentPos = endPos - overlapSize can move forward by only a few tokens. Text between the snap point and the forced forward index gets silently dropped.

Step 2: Semantic chunking on embedding similarity drops

Semantic chunking embeds sentences and cuts on cosine similarity drops — where meaning shifts, not where the token count hits a threshold:

function cosineSimilarity(vec1: number[], vec2: number[]): number {
  const dot = vec1.reduce((sum, val, i) => sum + val * vec2[i], 0);
  const mag1 = Math.sqrt(vec1.reduce((sum, val) => sum + val * val, 0));
  const mag2 = Math.sqrt(vec2.reduce((sum, val) => sum + val * val, 0));
  return mag1 > 0 && mag2 > 0 ? dot / (mag1 * mag2) : 0;
}
 
async function semanticChunker(
  text: string,
  embeddingFunction: (text: string) => Promise<number[]>,
  options: {
    similarityThreshold: number;
    minChunkSize: number;
    maxChunkSize: number;
  }
): Promise<TextChunk[]> {
  const sentences = text
    .split(/(?<=[.!?])\s+(?=[A-Z])/)
    .map(s => s.trim())
    .filter(s => s.length > 0);
 
  if (sentences.length === 0) return [];
 
  const embeddings = await Promise.all(sentences.map(embeddingFunction));
 
  const chunks: TextChunk[] = [];
  let currentGroup: string[] = [sentences[0]];
  let startIndex = 0;
 
  for (let i = 1; i < sentences.length; i++) {
    const similarity = cosineSimilarity(embeddings[i - 1], embeddings[i]);
 
    const currentGroupText = currentGroup.join(' ');
    const potentialSize = currentGroupText.length + sentences[i].length;
 
    const shouldSplit =
      (similarity < options.similarityThreshold &&
        currentGroupText.length >= options.minChunkSize) ||
      potentialSize > options.maxChunkSize;
 
    if (shouldSplit) {
      const chunkContent = currentGroup.join(' ');
      chunks.push({
        content: chunkContent,
        startIndex,
        endIndex: startIndex + chunkContent.length,
        metadata: { sourceId: 'doc-semantic', chunkIndex: chunks.length },
      });
      startIndex += chunkContent.length + 1;
      currentGroup = [sentences[i]];
    } else {
      currentGroup.push(sentences[i]);
    }
  }
 
  if (currentGroup.length > 0) {
    const chunkContent = currentGroup.join(' ');
    chunks.push({
      content: chunkContent,
      startIndex,
      endIndex: startIndex + chunkContent.length,
      metadata: { sourceId: 'doc-semantic', chunkIndex: chunks.length },
    });
  }
 
  return chunks;
}

The similarity threshold doesn't generalize across domains. A cutoff of 0.7 works for prose but over-segments dense technical manuals and financial tables. Profile the similarity distribution on your actual corpus before setting a hard number.

Step 3: Embedding model selection against real constraints

The embedding model choice should be driven by a measurable constraint, not benchmarks on someone else's dataset.

For air-gapped environments or p99 latency SLAs under 50ms, bge-small-en-v1.5 via ONNX Runtime was the only viable option on the projects I ran. For everything else, text-embedding-3-small with truncated dimensions (768 instead of 1536) cut storage cost in half with less than 2% Hit Rate degradation on my domain test set.

If you can't state the specific constraint that dictated your choice — latency, RAM footprint, cost per million tokens, or measured Hit Rate on your corpus — you picked arbitrarily.

Step 4: Measure retrieval with Hit Rate and MRR

Two metrics replace guesswork. Build a validation set of 50–200 representative queries with ground-truth chunk labels before running any experiments.

Hit Rate at k — what fraction of queries retrieved at least one correct chunk in the top k results:

Mean Reciprocal Rank at k — how high the first correct chunk ranks on average. Penalizes pipelines that retrieve the right chunk but bury it at position 4 or 5:

async function runEvaluation(
  retriever: { retrieve: (q: string, k: number) => Promise<TextChunk[]> },
  queries: string[],
  groundTruth: Map<string, Set<string>>,
  k: number = 5
): Promise<{ hitRate: number; mrr: number }> {
  let totalHit = 0;
  let totalReciprocalRank = 0;
 
  for (const query of queries) {
    const retrieved = await retriever.retrieve(query, k);
    const relevant = groundTruth.get(query) || new Set<string>();
 
    let rankOfFirst = 0;
    for (let i = 0; i < retrieved.length; i++) {
      const isRelevant = Array.from(relevant).some(
        gt => retrieved[i].content.includes(gt) || gt.includes(retrieved[i].content)
      );
      if (isRelevant) {
        totalHit++;
        rankOfFirst = i + 1;
        break;
      }
    }
    if (rankOfFirst > 0) totalReciprocalRank += 1 / rankOfFirst;
  }
 
  return {
    hitRate: totalHit / queries.length,
    mrr: totalReciprocalRank / queries.length,
  };
}

Hold the test queries and embedding model constant, swap only the chunker, and the delta in Hit Rate and MRR is the actual effect of your chunking decision. Without this, you're guessing.

On my own domain corpus (legal contracts, ~4,000 documents):

The reranker (a cross-encoder run over the top-20 candidates before generation) added the biggest single jump. It's also the most expensive — worth profiling your p99 latency budget before adding it.

Step 5: Architectural extensions worth evaluating in order

Once your baseline Hit Rate is above 70% and you have a working eval loop, these additions are worth testing in order of typical impact:

Hybrid retrieval (BM25 + dense vector, fused via RRF): Sparse lexical matching handles exact keyword queries that dense embeddings miss. Reciprocal Rank Fusion merges the two ranked lists without requiring calibrated score thresholds.

Hierarchical indexing: Embed at two granularities — section summaries and sentence-level chunks. Retrieve candidate sections first, then pass child chunks to the context window. Reduces irrelevant context without sacrificing recall.

Query rewriting: Use an LLM to generate a hypothetical ideal document (HyDE) or decompose multi-hop questions before searching. Adds latency; only worth it if your queries are consistently ambiguous or multi-part.

Context reranking: A cross-encoder (like bge-reranker-large) re-scores the top-20 candidates with full token-level attention. The largest accuracy gain, but ~150ms additional latency per query.

When semantic chunking is not worth the complexity

Semantic chunking adds embedding cost at ingestion time and requires threshold tuning. Skip it when:

• Documents are pre-segmented: FAQ databases, support tickets, API documentation, and code docstrings already have natural boundaries.
• The corpus is single-topic throughout: similarity rarely drops below any threshold, so the semantic chunker produces the same output as fixed-size.
• You have no eval benchmark yet: swapping chunkers without measuring Hit Rate and MRR gives you no signal. Establish the baseline first.

Common failure modes

Regex sentence splitters fail on technical text. Simple period-based splitting breaks on version numbers (v2.1.0), decimal values, code snippets, and abbreviations. Every downstream metric inherits that segmentation error silently.

String containment inflates eval scores. Evaluating chunk relevance with includes() can match boilerplate phrases shared across many chunks. Use token overlap (ROUGE-L) or exact identifier matching instead.

Embedding model version drift. Re-embedding documents with a newer model version without schema versioning corrupts vector space comparisons. Store the exact model identifier per row.