Guides/Retrieval architecture
Retrieval architecture cheat sheet
Retrieval quality dominates many RAG launches: if you fetch the wrong chunks, the generator confidently rehearses the wrong facts. Use this page to align with engineers on pipeline stages, pattern upgrades, and when extra complexity pays off.
End-to-end pipeline (canonical mental model)
Offline indexing runs batch or streaming; online path executes per query. Bottlenecks usually appear around chunk boundaries, embedding mismatch, and rerank latency — call those out explicitly in PRDs.
Pattern menu & tradeoffs
Start simple, measure recall@k and grounded answer rate, then layer hybrid search or reranking where queries fail — not before you have a labeled failure set.
| Pattern | Core idea | Upside | Watch-outs |
|---|---|---|---|
| Naive vector | Embed chunks; nearest neighbors to query | Fast to ship; few moving parts | Lexical mismatch; duplicate chunks hurt recall |
| Hybrid (sparse + dense) | BM25-style + vectors fused or reranked | Better recall on keywords & SKUs | More infra; tuning fusion weights |
| + Cross-encoder rerank | Retrieve wide, rerank top-K with heavier model | Precision jump on ambiguous queries | Latency & cost at query time |
| Agentic retrieval | Model plans sub-queries, tools, iterative fetch | Handles multi-hop reasoning | Fragile loops; hardest to eval |
Complexity vs illustrative quality uplift
Bars encode engineering + ops burden versus expected retrieval lift — agentic retrieval breaks this curve when tool loops mis-fire, so budget eval harnesses before chasing autonomy.
Complexity vs illustrative quality uplift
Quality uplift assumes tuned baselines — agentic can regress if loops go wrong.
PM checkpoints
- Define freshness SLAs for corpora that feed customer-visible answers.
- Instrument citation coverage — percent of answers that reference retrieved passages when policy demands it.
- Pair retrieval upgrades with regression suites so embedding or reranker swaps do not silently regress recall.