Self-hosted RAG: GDPR-compliant search

Language models are impressive – but they don't know your internal documents. RAG (Retrieval-Augmented Generation) closes that gap: it searches your knowledge base and hands the model the relevant passages as context. This guide builds a fully self-hosted RAG system – with Qdrant, BGE-M3, reranking and a local LLM. So every document stays in-house.

What is RAG?

A language model answers from its training knowledge – it doesn't know your contracts, manuals and tickets. You could retrain it, but that's expensive, slow and impractical for data that keeps changing. RAG flips the approach: for every question, it first finds the topically relevant passages from your documents and hands them to the model as context. The model then phrases its answer purely from that grounded material.

The core is semantic search: texts are translated into vectors (sequences of numbers) that capture their meaning. Similar meaning means a small distance in vector space – so the system finds the right passage even when the question uses different words than the document.

Why self-host & GDPR

A RAG system brings together exactly the most sensitive data: your internal documents and the concrete questions your staff ask. If those go to an external cloud API, you hand over trade secrets and personal data. Self-hosted, everything stays local instead – the simplest, cleanest basis for GDPR compliance, confidentiality and auditability. The language model is a local vLLM server, just like the one we set up in Proxmox 9 with a GPU for a vLLM VM.

The building blocks

A RAG stack consists of a few clearly separated components – all open source and self-hostable:

1.Start the vector database

Qdrant runs as a lightweight container. Important: Qdrant ships without authentication by default – be sure to set an API key, otherwise anyone on the network can read and delete your index.

# Secure Qdrant with an API key (port 6333 = REST/dashboard, 6334 = gRPC)
docker run -d --name qdrant \
  -p 6333:6333 -p 6334:6334 \
  -e "QDRANT__SERVICE__API_KEY=your-qdrant-key" \
  -v "$(pwd)/qdrant_storage:/qdrant/storage:z" \
  qdrant/qdrant

2.Set up the environment

A fresh Python environment and the required libraries – again with the fast package manager uv:

uv venv --python 3.12 --seed
source .venv/bin/activate

# Vector-DB client, embedding/reranker models, OpenAI client, PDF parser
uv pip install qdrant-client sentence-transformers openai pypdf

3.Collection & model

We connect to Qdrant and create a collection – the container for our vectors. BGE-M3 produces 1024-dimensional vectors; for the distance metric we use cosine similarity.

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams
from sentence_transformers import SentenceTransformer

qdrant = QdrantClient(url="http://localhost:6333", api_key="your-qdrant-key")
model = SentenceTransformer("BAAI/bge-m3")   # local, multilingual

# Create the collection: 1024 dimensions, cosine distance
if not qdrant.collection_exists("knowledge_base"):
    qdrant.create_collection(
        collection_name="knowledge_base",
        vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
    )

4.Prepare documents

Documents are split into overlapping chunks. Chunks that are too large dilute the meaning; too small and they tear apart the context. An overlap keeps sentences at the boundary from getting lost.

from pathlib import Path
from pypdf import PdfReader

def load_text(path: Path) -> str:
    if path.suffix.lower() == ".pdf":
        return "\n".join(p.extract_text() or "" for p in PdfReader(str(path)).pages)
    return path.read_text(encoding="utf-8")

# Split into ~800-word chunks with 120 words of overlap
def chunk(text: str, size: int = 800, overlap: int = 120):
    words = text.split()
    step = size - overlap
    for i in range(0, len(words), step):
        yield " ".join(words[i:i + size])

5.Index

Now we create a vector for each chunk and store it together with metadata (text and source) in Qdrant. The source is worth its weight in gold later: for citations, for filters and for the right to erasure.

import uuid
from qdrant_client.models import PointStruct

# Load, chunk and index every file in a folder
def index(folder: str = "documents"):
    points = []
    for file in Path(folder).glob("**/*"):
        if file.is_dir():
            continue
        for piece in chunk(load_text(file)):
            vector = model.encode(piece, normalize_embeddings=True)
            points.append(PointStruct(
                id=str(uuid.uuid4()),
                vector=vector.tolist(),
                payload={"text": piece, "source": file.name},
            ))
    qdrant.upsert(collection_name="knowledge_base", points=points)
    print(f"{len(points)} chunks indexed.")

6.Vector search

The question is embedded with the same model and matched against the index. Qdrant returns the k most similar passages – deliberately a few more than we ultimately need, because the reranker in the next step sifts out the truly best ones.

def search(question: str, k: int = 8):
    qv = model.encode(question, normalize_embeddings=True).tolist()
    hits = qdrant.query_points(
        collection_name="knowledge_base", query=qv, limit=k, with_payload=True,
    ).points
    return [(h.payload["text"], h.payload["source"], h.score) for h in hits]

7.Reranking

Vector search is fast but coarse. A cross-encoder reads the question and a passage together and judges the fit far more accurately. That pushes the truly relevant passages to the top – half the battle for good answers.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

def rerank(question: str, candidates, top_n: int = 4):
    pairs = [(question, text) for text, _, _ in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(scores, candidates), key=lambda x: x[0], reverse=True)
    return [c for _, c in ranked[:top_n]]

8.Generate the answer

The best passages go into the prompt as context. A clear instruction makes the model answer only from the grounded material and cite the source – which guards against hallucinations. The model is the local vLLM server via its OpenAI-compatible API.

from openai import OpenAI

llm = OpenAI(base_url="http://localhost:8000/v1", api_key="your-secret-key")

def answer(question: str) -> str:
    context = rerank(question, search(question))
    sources = "\n\n".join(f"[{s}] {t}" for t, s, _ in context)
    prompt = (
        "Answer the question using only the context below. "
        "Cite the source in square brackets. If the context is insufficient, say so.\n\n"
        f"Context:\n{sources}\n\nQuestion: {question}"
    )
    reply = llm.chat.completions.create(
        model="llama3.1-8b",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.1,
    )
    return reply.choices[0].message.content

print(answer("How long do we retain incoming invoices?"))

9.Expose as an API

A thin FastAPI wrapper makes the system usable for applications, chat interfaces or the intranet – and runs best as a container itself:

from fastapi import FastAPI
from pydantic import BaseModel
from rag import answer

app = FastAPI()

class Query(BaseModel):
    question: str

@app.post("/ask")
def ask(q: Query):
    return {"answer": answer(q.question)}

The dependencies go into a requirements.txt, the service into a lean image:

qdrant-client
sentence-transformers
openai
pypdf
fastapi
uvicorn[standard]

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY rag.py ingest.py app.py .
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8800"]

A docker-compose.yml starts Qdrant and the RAG API together as one stack – with automatic restart and persistent storage:

services:
  qdrant:
    image: qdrant/qdrant:latest
    restart: unless-stopped
    environment:
      - QDRANT__SERVICE__API_KEY=your-qdrant-key
    volumes:
      - ./qdrant_storage:/qdrant/storage
  rag-api:
    build: .
    restart: unless-stopped
    depends_on:
      - qdrant
    ports:
      - "8800:8800"
    volumes:
      - ./documents:/app/documents
      - ./models:/models

docker compose up -d --build

You populate the knowledge base once as a one-off container run using the same image – a tiny ingest.py calls the indexing function from rag.py:

# One-off script: reads the "documents" folder and indexes everything
from rag import index

if __name__ == "__main__":
    index("documents")

# Drop your files into ./documents, then start a one-off container
docker compose run --rm rag-api python ingest.py

Improve quality

• Tune the chunking: size and overlap are the most powerful levers. Rule of thumb: 500–1000 words, 10–20% overlap – ideally cut at paragraphs rather than mid-sentence.
• Hybrid search: BGE-M3 also produces sparse (keyword-like) vectors. Combining dense and sparse search finds both meaning and exact terms (part numbers, proper names).
• Don't skip reranking: it's the biggest quality gain per line of code.
• Metadata filters: filter by department, tenant or date – this sharpens the hits and enforces access rights.
• Show sources: citations make answers verifiable and build trust.
• Evaluate: a small set of real questions with expected sources quickly shows whether a change is actually better.

GDPR & security

Self-hosting provides the foundation – the implementation makes the difference:

• Data sovereignty: embeddings, vector DB and LLM all run in your own network. No question, no document leaves the building.
• Access control: mirror the permissions from your source systems in the metadata and enforce them as filters at query time – users only see what they're allowed to.
• Right to erasure: because every chunk knows its source, a document can be removed from the index precisely and completely.

from qdrant_client.models import Filter, FieldCondition, MatchValue

# Remove all vectors of a document by its source
qdrant.delete(
    collection_name="knowledge_base",
    points_selector=Filter(must=[
        FieldCondition(key="source", match=MatchValue(value="personnel_file_smith.pdf")),
    ]),
)

Troubleshooting

Frequently asked questions

Sources

External sources, as of August 2026 (open in a new tab):

• Qdrant – local quickstart (Docker, collections, API key)
• BAAI – BGE-M3 (multilingual embedding model)
• BAAI – bge-reranker-v2-m3 (cross-encoder)
• vLLM – pooling models (embeddings & reranking)
• Qdrant – official Docker image

This article is for general information. Code, model and version details are examples and must be adapted to your environment. As of August 2026.