Building production RAG on local LLMs
What it actually takes to run retrieval-augmented generation on open-source models inside your own estate: model selection, hardware sizing, retrieval design, evaluation, and the operational practices that keep it honest.
Local RAG is no longer a research project. The combination of strong open-weights models, mature serving stacks, and well-understood retrieval patterns means that on-premise retrieval-augmented generation is now a real production option, and for regulated workloads often the only one. This article walks through the choices that actually matter.
When local RAG is the right answer
Local RAG earns its keep when one of three things is true: data classification rules out hosted providers, latency budgets cannot tolerate the round-trip to a hosted API, or the cost curve at scale tips in favour of self-hosting. For governments and regulated financial institutions, classification rules out hosted providers more often than people expect, and the architecture must be designed around that constraint from day one rather than bolted on later.
If none of those three conditions hold, a hosted frontier model is almost always the faster path to value. Use local RAG where it pays for the operational overhead.
Model selection: capacity vs. footprint
The interesting choices today sit between 8B and 70B parameter open-weights models. Below 8B, you trade reasoning quality for footprint in ways that hurt regulated workflows. Above 70B, you start paying for capacity you may not need if your retrieval is doing its job.
Quantisation is the lever that closes the gap. INT4 / INT8 quantisation on a well-chosen base model often delivers 90% of the original behaviour at a fraction of the GPU cost. The honest test is your evaluation harness, not a leaderboard.
Retrieval is where the system is won or lost
Most failed RAG systems fail at retrieval, not at generation. A model cannot answer correctly from a wrong passage; it can only fabricate confidently.
Hybrid retrieval (sparse BM25 or SPLADE plus dense vectors) almost always beats either alone. A well-tuned re-ranker on the top 50 candidates is the single highest-leverage component most teams skip. Domain-specific embeddings, where you have the data to fine-tune them, are worth the effort for specialist corpora.
Evaluation: golden sets and continuous regression
If you cannot measure retrieval recall and answer quality on a held-out golden set, you cannot improve the system. Build the harness before the demo.
Two metrics matter early: recall@k for retrieval and a calibrated answer-quality score for generation. Run both on every model change and every corpus update; treat regressions as you would treat a failing test.
Deployment: vLLM, observability, guardrails
vLLM has become the default serving stack for open-weights LLMs in production for good reasons: paged attention, continuous batching, and high throughput out of the box. Pair it with OpenTelemetry traces for the full retrieval-and-generation path, and a guardrail layer that rejects responses missing required citations.
Operationally, treat the model as a versioned dependency. Pin versions in production, run shadow traffic for replacements, and never roll out a model change without a rolled-up evaluation report.
What we tell clients
Build the evaluation harness first. Pick a sensible default stack (vLLM + a strong 8B–70B open-weights model + hybrid retrieval + re-ranker) and resist the urge to optimise prematurely. Spend the time you save on the corpus: chunking, metadata, and the boring work of getting the documents right is what separates a useful system from a demo.
Where we apply this in practice
Working on this in your own organisation?
If any of this maps to a programme you are running, we would be glad to compare notes.