Running RAG locally: from embedding to answer

Along the route of running language models locally, this article sits in the phase of connecting and enriching. Once a local model is installed and running properly, the need arises immediately to feed the model your own current or confidential source files. For a solid grounding in the required compute power and memory bandwidth, consult the article on hardware for local LLMs, which explains the balance between VRAM and CPU memory.

Retrieval-Augmented Generation (RAG) resolves the two fundamental limitations of every language model: a static knowledge cutoff and a tendency to hallucinate. By retrieving targeted passages from your own documents and passing them along in the context, the model bases its answer directly on verifiable sources. In this complete guide we work through the full chain on your own hardware: from reading and splitting raw text to vector representation, semantic search, context injection, and the final synthesis.

1. The anatomy of a local RAG pipeline

A RAG architecture consists of two separate phases: the ingestion phase (offline preparation) and the query phase (real-time retrieval and generation). In a cloud environment these steps are often outsourced to external APIs, but on a local system every component runs directly on your own machine. That guarantees full data sovereignty.

During the ingestion phase, documents such as PDFs, Markdown notes, or Word files are read in, stripped of noise, split into semantically coherent fragments (chunks), and converted into dense numerical vectors (embeddings). These vectors are indexed in a local vector store. In the query phase, the embedding model converts the user's question into the same vector space, after which the database selects the best-matching text fragments based on cosine similarity or dot product.

The generative model is then presented with a combined prompt: the system instruction, the retrieved text fragments as evidence, and the original question. Here the model acts primarily as an intelligent syntax and reasoning engine that summarizes the context and turns it into a natural answer, without leaning on its own (possibly outdated) parametric memory.

2. Document parsing and chunking strategies

The quality of the generated answers stands or falls with the precision of the chunking step. If a text fragment is too large, the relevant information drowns in irrelevant context, which overloads the attention mechanisms of smaller local models. If the fragment is too small, the text loses its grammatical and semantic coherence.

For structured Dutch text, a recursive character splitter with semantic separators delivers the best results. The algorithm first splits on double line breaks (paragraphs), then on single line breaks, then on sentences, and only as a last resort on individual words. An overlap between consecutive fragments prevents crucial sentences from being cut in half exactly at a break point.

Below is a concrete Python example in which raw text is split using structural anchors:

def chunk_tekst(tekst: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    woorden = tekst.split()
    chunks = []
    stap = chunk_size - overlap
    for i in range(0, len(woorden), stap):
        segment = " ".join(woorden[i:i + chunk_size])
        if segment:
            chunks.append(segment)
    return chunks

# Voorbeeld met documentverwerking
brontektst = "Lokale AI waarborgt privacy door data binnen het eigen netwerk te houden."
fragmenten = chunk_tekst(brontektst, chunk_size=200, overlap=30)

3. Embeddings: from text to compact vectors

An embedding model converts text fragments into a multidimensional vector that captures semantic meaning mathematically. Texts that are similar in meaning end up close together in the vector space, regardless of whether they contain exactly the same keywords.

For a Dutch-language RAG application, the language choice of the embedding model is crucial. Many standard models are trained primarily on English data and lose a lot of nuance on Dutch grammatical structures. Consult the detailed comparison on the right embedding model for Dutch documents to determine when a multilingual model such as BGE-M3 or a Dutch-specific model performs best.

Embedding models require considerably less compute than generative models. A model of 500 million parameters runs effortlessly on a CPU or requires less than 1.5 GB of VRAM on a graphics card. When configuring, pay close attention to dimension size: 384 dimensions (as in MiniLM) require less memory and compute during indexing, while 1024 dimensions (as in BGE-M3) hold considerably more depth and cross-connections between complex case files.

4. Storage and semantic search in the vector database

Once text fragments have been converted into vectors, they have to be stored and searched efficiently. For that we use a specialized vector database that relies on HNSW indexing (Hierarchical Navigable Small World) to find the nearest neighbors among hundreds of thousands of fragments within milliseconds.

In a local setup, Qdrant and Chroma are the most common choices. If you want to get straight to installing and configuring them, practical instructions can be found in the guide on setting up a local vector database with Qdrant and Chroma, which works through making collections persistent on disk step by step.

Retrieving vectors usually happens on the basis of cosine similarity. The formula calculates the angle between the query vector and the stored document vectors:

import numpy as np

def cosine_similarity(v1: np.ndarray, v2: np.ndarray) -> float:
    dot_product = np.dot(v1, v2)
    norm_v1 = np.linalg.norm(v1)
    norm_v2 = np.linalg.norm(v2)
    if norm_v1 == 0 or norm_v2 == 0:
        return 0.0
    return float(dot_product / (norm_v1 * norm_v2))

5. Retrieval strategies: pure vector search versus hybrid and reranking

Purely semantic search has a well-known weak point: it recognizes conceptual relationships excellently, but performs poorly on exact keywords, article numbers, case codes, or personal names. If a document asks for "Invoice 2026-F44", a vector search looks for the concept of a bill rather than that specific code.

To solve this, an advanced RAG setup combines semantic search with traditional keyword search (BM25). This hybrid approach yields two result lists, which are merged with Reciprocal Rank Fusion (RRF). A lightweight cross-encoder (reranker) then assesses the top 20 results to select the 3 to 5 most relevant fragments for the prompt. For a theoretical and practical trade-off between these methods, the overview of embeddings, rerankers, or hybrid retrieval offers in-depth comparative benchmarks.

6. Context injection and prompt construction

Once the relevant text fragments have been selected, they are combined into the generative model's context window. The way this context is structured has a direct influence on the model's reasoning ability. When sources are presented unordered, the 'lost in the middle' effect often occurs, where the model ignores information in the middle of the prompt.

A proven prompt template for local models uses strict delimiters and clear instructions to answer solely on the basis of the supplied context:

Je bent een betrouwbare assistent die vragen beantwoordt op basis van documenten.
Beantwoord de onderstaande vraag UITSLUITEND met behulp van de meegeleverde bronfragmenten.
Als het antwoord niet in de bronnen staat, zeg dan expliciet: "Op basis van de beschikbare documenten kan ik deze vraag niet beantwoorden."

=== BRONNEN START ===
[Bron 1 - Documentatie.pdf (pagina 4)]
Lokale RAG-architecturen vereisen geen externe netwerkverbindingen voor data-opslag.

[Bron 2 - Beleid2026.docx (sectie 2.1)]
Alle interne verslagen dienen lokaal te worden verwerkt onder Q4_K_M kwantisatie.
=== BRONNEN EINDE ===

Vraag: Welke kwantisatie moeten interne verslagen volgens het beleid gebruiken?
Antwoord:

When configuring the generative model, the memory footprint of the context window plays a major role. To keep the model running smoothly without stalls, a compact format is essential. The article on quantization methods for local models explains how techniques such as 4-bit and 5-bit GGUF compression ensure that both the model and the extended RAG context fit within the VRAM budget.

7. The complete integration in Python

Below is a compact but functional Python script that connects the entire chain: generating a query embedding, retrieving context from a local vector store, and calling a local language model through an OpenAI-compatible interface (such as Ollama or the llama.cpp server):

import requests
import json

OLLAMA_API = "http://localhost:11434/api"

def haal_embedding(tekst: str) -> list[float]:
    payload = {"model": "bge-m3", "prompt": tekst}
    response = requests.post(f"{OLLAMA_API}/embeddings", json=payload)
    return response.json()["embedding"]

def genereer_rag_antwoord(vraag: str, context_fragmenten: list[str]) -> str:
    context_tekst = "\n\n".join([f"- {frag}" for frag in context_fragmenten])
    system_prompt = (
      "Beantwoord de vraag feitelijk op basis van de onderstaande context.\n\n"
      f"Context:\n{context_tekst}"
    )
    
    payload = {
      "model": "qwen2.5:7b-instruct-q4_k_m",
      "messages": [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": vraag}
      ],
      "stream": False,
      "options": {"temperature": 0.1, "num_ctx": 4096}
    }
    
    res = requests.post("http://localhost:11434/v1/chat/completions", json=payload)
    return res.json()["choices"][0]["message"]["content"]

# Uitvoering
gebruikersvraag = "Wat is het stroomverbruik van de server in ruststand?"
opgehaalde_chunks = [
  "Metingen tonen aan dat de lokale server in ruststand exact 14 Watt verbruikt.",
  "Onder volledige belasting met 2 actieve streams stijgt het vermogen naar 185 Watt."
]

antwoord = genereer_rag_antwoord(gebruikersvraag, opgehaalde_chunks)
print(antwoord)

8. Quality control, fact-checking, and error diagnosis

Even with a carefully built RAG chain, errors can occur. Common failure modes are:

• Retrieval failure: The relevant information is present in the documents, but does not appear in the top-k results because of a poor search term or suboptimal chunking.
• Context confusion: Two contradictory sources are placed in the context at the same time, causing the model to make an arbitrary choice or to combine both.
• Hallucination despite context: The model ignores the explicit instruction and falls back on pretrained assumptions when a source is vaguely worded.

To verify that the output genuinely matches the underlying sources, a systematic evaluation method is essential. See the article on fact-checking AI answers for practical techniques to compare claims automatically against source documents and to detect inconsistencies.

9. Privacy, network isolation, and data security

The main argument for a local RAG architecture is the guarantee that confidential data never leaves your own hardware. With cloud-based RAG services, documents, queries, and embeddings pass through several external servers and storage locations.

In a local setup, the entire data flow stays inside the local network or even inside a single machine. No telemetry is collected and no external API key is needed. If you are curious about the legal and operational safeguards of this approach, the overview of working with AI in a privacy-friendly way explains how a strictly local model meets stringent data protection requirements.

In a home setting or a small office too, a walled-off environment prevents private data and personal records from accidentally getting out. Practical measures for isolating systems locally are covered in the guide on safe AI use in a home environment, which focuses on network segmentation and access rights.

Summary and next steps

A locally running RAG pipeline turns a generic open-source language model into a specialized knowledge assistant that reasons exclusively about your own files. Careful document parsing, a suitable embedding model, persistent vector storage, and sharp context injection produce a reliable system that responds quickly and offers full privacy.

After setting up the base pipeline, the next step lies in automating evaluations and adding specialized filters. That turns the local RAG infrastructure into a robust, scalable foundation for day-to-day professional document management.