Large language models are trained on public data up to a cut-off date. They know nothing about your price list, your HR policy or last week's project report. Ask them anyway and they may produce a confident, wrong answer.

Retrieval-augmented generation (RAG) fixes this. Before the model answers, the system finds the most relevant passages from your documents and gives them to the model as context. The model then answers from that context — and can show which document each fact came from.

How RAG works, step by step

  1. Collect documents — PDFs, web pages, wiki articles, support tickets.
  2. Split them into chunks of a few hundred words, keeping headings so each chunk still makes sense on its own.
  3. Create embeddings — numeric vectors that capture each chunk's meaning — and store them in a vector index (for example PostgreSQL with pgvector).
  4. At question time, embed the question, find the closest chunks, and often combine this with keyword search so exact terms such as product codes are not missed.
  5. Ask the model to answer using only the retrieved chunks, and to say "I don't know" when they do not contain the answer.
  6. Return the answer with citations so the reader can check the source.

Why it beats fine-tuning for most knowledge tasks

  • Always current: update a document and the next answer reflects it — no retraining.
  • Traceable: every answer can point to its source.
  • Access-aware: you can filter retrieved chunks by the user's permissions.
  • Cheaper: indexing documents costs far less than training a model.

Fine-tuning is still useful for teaching a model a style or a format, but not for keeping facts up to date.

Common mistakes

  • Chunks that are too small or too large. Tiny chunks lose context; huge ones dilute relevance. Start around 300–500 words with some overlap.
  • Ignoring tables and scanned PDFs. Run OCR and keep table structure, or key numbers will never be found.
  • Vector search only. Pure semantic search misses part numbers and acronyms. Hybrid search (keywords + vectors) is more reliable.
  • No "I don't know" rule. Without it the model will fill gaps with guesses.
  • No evaluation set. Keep 30–50 real questions with known answers and re-test after every change.

A quick quality checklist

  • Does every answer cite at least one source?
  • Does the assistant refuse when the documents do not contain the answer?
  • Are confidential documents only retrievable by people allowed to see them?
  • Is answer accuracy measured on a fixed question set?

Key takeaways

RAG grounds AI answers in your own content, keeps them current and makes them checkable. Most of the quality comes from good document preparation, hybrid retrieval and honest "I don't know" behaviour — not from the choice of model.