Adding local re-ranking to your RAG search pipeline

In the fixed development route of local AI — choose → install → use → connect → manage/serve — we are at the step connect. Anyone who has documents ready in a vector store and notices that search results are just not relevant enough needs a qualitative intermediate step. Before extending the setup, it is wise to check that the underlying infrastructure is solid; consult which hardware is suitable for local language models to verify that your machine has enough memory bandwidth for multiple processing steps.

Many local RAG systems (Retrieval-Augmented Generation) run into the same frustration: the vector database retrieves just the wrong blocks of text. A semantic vector search (bi-encoder) can select the twenty most likely candidates from tens of thousands of paragraphs at blazing speed, but it often lacks the fine-grained precision to determine which three fragments really contain the answer. By placing a local re-ranker (cross-encoder) between the search step and the generative language model, context quality rises considerably without any need for heavier generation models.

Why bi-encoders fall short in RAG

A standard search step in a vector database uses what is called a bi-encoder. Here document chunks and search queries are converted into compact vectors independently of each other. The advantage is compute speed: the vectors of all documents are calculated once and stored. During a search the database only has to compute the cosine similarity between the vector of the user's question and the pre-stored document vectors.

This separation does carry a fundamental limitation. Because the query and the document fragment never "meet" during the mathematical transformation, complex nuances such as negations, specific product codes, or conditional sentences can get lost in the compressed vector space. To keep a clear view of how these vector concepts relate to other ideas, the complete list of AI and machine learning terms offers a concise reference framework for the underlying terminology.

A cross-encoder takes a different approach. It analyzes the query and the candidate text block simultaneously through full cross-attention. Every word in the question interacts with every word in the document. That lets the model determine whether a paragraph genuinely answers the specific question, rather than merely establishing that both texts concern the same general theme.

The architecture of a two-stage search pipeline

Cross-encoders are far too compute-intensive to unleash directly on a complete collection of tens of thousands of documents. That is why we combine the best of both worlds in a classic two-stage architecture:

In step one the vector database handles the coarse filtering. We do not retrieve the usual top 3 or top 5 fragments but widen the search window to, say, 25 to 50 candidate chunks (high recall). In step two the local re-ranker judges these 25 fragments one by one in interaction with the query. The re-ranker assigns each fragment a relevance score between 0 and 1. We then select only the best 3 to 5 fragments with the highest score (high precision) and pass those on to the local LLM.

To feed this two-stage pipeline you need a reliable index; the article on setting up a local vector database with Qdrant or Chroma shows how to configure such storage systems and prepare them for larger retrieval volumes.

Choosing local models and libraries

For local re-ranking there are two dominant approaches: lightweight ONNX models through optimized CPU runtimes, or full transformer models through PyTorch or C++ backends.

FlashRank is the most accessible solution. It runs entirely on CPU through ONNX Runtime, requires no heavy PyTorch dependencies, and has standard models of only 4 to 90 MB (such as ms-marco-TinyBERT-L-2-v2 at roughly 4 MB, ms-marco-MiniLM-L-12-v2 at roughly 34 MB, and ms-marco-MiniLM-L-6-v2 at roughly 83 MB). ms-marco-TinyBERT-L-2-v2 and bge-reranker-large in ONNX format). Latency usually sits between 15 and 40 milliseconds for 20 text fragments on a standard consumer CPU.

BGE-Reranker (BAAI), such as bge-reranker-v2-m3 or bge-reranker-large, is the gold standard when multilingual coverage and maximum precision are required. The M3 variant supports more than 100 languages and handles contexts up to 8192 tokens. This model runs excellently on a local GPU through HuggingFace Sentence Transformers or through ONNX with DirectML/CUDA acceleration.

Anyone running their pipeline on a Linux environment can consult the guide on setting up local language models under Linux for maximum performance and to configure CUDA or ROCm drivers properly right away.

Practical example with Python and FlashRank

The script below demonstrates how we reorder a list of raw search results from an arbitrary data source with FlashRank. First install the minimal runtime through pip install flashrank.

from flashrank import Ranker, RerankRequest

# Initialiseer de lichtgewicht ranker (laadt ONNX-model lokaal)
ranker = Ranker(model_name="ms-marco-TinyBERT-L-2-v2", cache_dir="./opt/models")

query = "Wat is het maximale vermogen van de laadpaal bij piekbelasting?"

# Gesimuleerde ruwe opbrengst uit de eerste zoekfase (top-k = 5)
ruwe_chunks = [
    {
        "id": 1,
        "text": "De laadpaal kan worden aangesloten op een standaard 3-fase "
                "aansluiting van 3x25A in de meterkast."
    },
    {
        "id": 2,
        "text": "Tijdens dynamische pieklastbeperking schaalt de laadpaal het "
                "maximale laadvermogen automatisch terug naar 11 kW."
    },
    {
        "id": 3,
        "text": "Onderhoud aan het laadstation dient jaarlijks te worden "
                "uitgevoerd door een gecertificeerd installateur."
    },
    {
        "id": 4,
        "text": "Het nominale piekvermogen bedraagt 22 kW onder ideale "
                "omstandigheden zonder netbeperking."
    }
]

# Stel de re-ranking vraag samen
rerank_request = RerankRequest(query=query, passages=ruwe_chunks)
resultaten = ranker.rerank(rerank_request)

# Toon de gerangschikte uitkomst
for rang, item in enumerate(resultaten, start=1):
    score = item["score"]
    tekst = item["text"]
    chunk_id = item["id"]
    print(f"Rang {rang} (Score: {score:.4f}) [ID {chunk_id}]: {tekst}")

In this test the vector searcher often sees ID 4 as most relevant because of the word "peak power". The cross-encoder, however, immediately recognizes that the asker is looking specifically for behavior under peak load, which pushes ID 2 to the top position.

Integration with SentenceTransformers and PyTorch

For heavier setups with a dedicated GPU, the sentence-transformers library offers direct access to state-of-the-art models such as BAAI/bge-reranker-large.

import torch
from sentence_transformers import CrossEncoder

device = "cuda" if torch.cuda.is_available() else "cpu"
model_naam = "BAAI/bge-reranker-v2-m3"

# Laad de cross-encoder
cross_encoder = CrossEncoder(model_naam, max_length=512, device=device)

zoekvraag = "Welke opzegtermijn geldt voor een contract van onbepaalde duur?"
document_kandidaten = [
    "Contracten voor bepaalde tijd eindigen van rechtswege na de termijn.",
    "Bij een overeenkomst voor onbepaalde tijd geldt een wettelijke "
    "opzegtermijn van minimaal een kalendermaand.",
    "Facturen dienen binnen veertien dagen na dagtekening te worden voldaan."
]

# Bouw paren van (vraag, document)
paren = [[zoekvraag, doc] for doc in document_kandidaten]

# Bereken scores
scores = cross_encoder.predict(paren)

# Sorteer op basis van score in aflopende volgorde
geordend = sorted(
    zip(scores, document_kandidaten),
    key=lambda x: x[0],
    reverse=True
)

for score, doc in geordend:
    print(f"Score: {score:+.4f} -> {doc}")

The cross-encoder returns a logit score. The higher the number, the stronger the correlation between the question and the specific paragraph.

Impact on latency, memory, and quantization

Adding a re-ranker introduces an extra computation between the database and answer generation. We have to weigh this overhead carefully against the gain in quality.

Re-ranking 25 text blocks with a model of 300 million parameters on a CPU takes 40 to 100 milliseconds on average. On a modern Apple Silicon Mac (M1/M2/M3) or a system with a discrete Nvidia card this latency is negligible (< 20 ms). That is more than offset by the time saved at the LLM itself: because we need to send fewer context blocks to the generation model, the language model saves tens to hundreds of milliseconds of prompt processing time.

Anyone running on compact machines with limited working memory can shrink the model format further; read how quantization helps run AI models efficiently on local hardware without any appreciable loss of accuracy.

Illustrative scenario in a Dutch context

Dutch business and legal texts often contain specific compound words and formal phrasing, which can make it hard for a bi-encoder to capture the context and nuances properly. To make the workings and added value of local re-ranking on Dutch-language data clear, we look at an illustrative scenario involving complex search queries about policy documents.

Suppose that in an illustrative worked example without re-ranking (vector search alone with a general multilingual embedding model) the top-1 position contains the exact answer fragment in around sixty percent of cases. By reordering an initial selection of 20 fragments with bge-reranker-v2-m3 top-1 precision in such a scenario could rise to roughly ninety percent.

The quality of the first search stage nevertheless remains decisive for the eventual outcome; read the overview on choosing the right embedding model for Dutch texts to keep relevant documents from dropping out of the selection before the re-ranking stage. After all, a re-ranker can only judge fragments that the first selection brought in. The guide on getting language models to perform better in Dutch also helps keep the final answer phrasing tight and error-free.

Privacy and data protection

A crucial advantage of a fully local re-ranking solution is that document fragments and user questions never leave your own device or local network at any point. Many cloud-based re-ranking APIs require every raw text block to be processed through external servers. By hosting the vector database, the cross-encoder, and the generative model locally, all sensitive business information stays within your own infrastructure; see the guidelines on working with AI models in a privacy-friendly way for additional measures around data storage and logs.

Common mistakes and failure modes

Although re-ranking greatly improves the reliability of a RAG pipeline, there are specific pitfalls to keep in mind:

• Initial recall set too narrow: If the vector database retrieves only 5 candidates in the first stage, the re-ranker can no longer recover a missed relevant fragment. Always set the initial search stage to at least 20 to 40 documents.
• Exceeding the cross-encoder's token limits: Many re-rankers have a context limit of 512 tokens for the combined length of question and passage. Make sure your document chunks are no longer than 350 to 400 tokens, so the query always fits alongside them.
• Language incompatibility: Models trained exclusively on English-language datasets (such as pure MS-MARCO checkpoints) perform poorly on Dutch compound terms. Always choose a multilingual variant such as BGE-v2-M3 or Qwen2-Rerank.
• Blind trust in raw score cutoffs: Re-ranking scores are relative relevance indicators, not absolute truth values. Prefer a top-k selection (for instance the best 3 results) over a hard minimum threshold score.

Integration into a complete local RAG stack

When we bring all the components together, the result is a robust search chain that performs more consistently than traditional vector searchers. The flow runs linearly:

1. The user asks a question through the interface or API.
2. The vector index retrieves the 30 most similar chunks (recall stage).
3. Optionally: a BM25 search retrieves 30 keyword matches (hybrid search).
4. The re-ranker judges all unique candidates and assigns scores.
5. The best 3 to 4 text fragments are combined as context in the system prompt.
6. The local language model formulates a factual and concise answer.

This structure lowers the risk of hallucinations considerably, because the generation model is fed only with fragments whose relevance has been verified at the word level.