Fine-Tuning and RAG

Try prompting and RAG first

Most "make the model know our stuff" problems are solved with retrieval-augmented generation (RAG), not fine-tuning. Fine-tuning changes behaviour and style; it is a poor way to add facts that change.

RAG in five steps

1. Chunk your documents (for example 300–800 tokens with some overlap).
2. Embed each chunk with an embedding model into a vector.
3. Store vectors in a vector database (pgvector in PostgreSQL, Qdrant, Chroma).
4. Retrieve the most similar chunks for each question.
5. Generate an answer with the chunks in the prompt, asking the model to cite them and say when the answer is not present.

-- pgvector
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (id bigserial PRIMARY KEY, doc_id int, body text, embedding vector(768));
SELECT body FROM chunks ORDER BY embedding <=> :question_embedding LIMIT 5;

Combine vector search with keyword search (hybrid) for names, codes and exact terms.

When fine-tuning helps

• A consistent output format or style across thousands of requests.
• A specialised task where a small fine-tuned model can replace a larger general one.
• Domain language the base model handles poorly.

LoRA

Low-Rank Adaptation trains small adapter matrices instead of all weights, cutting memory and cost dramatically. QLoRA combines it with 4-bit quantisation so mid-sized models can be fine-tuned on a single GPU. Libraries such as Hugging Face PEFT, TRL and Unsloth make this approachable.

Data quality decides results

A few hundred to a few thousand high-quality, consistent examples usually beat a large messy dataset. Hold out a test set, and compare the fine-tuned model against the base model plus good prompting.

Exercise

Build a RAG assistant over your project documentation using pgvector and a local or hosted model: chunk, embed, store, retrieve and answer with citations. Evaluate it on 20 questions, including some the docs do not answer.

Check yourself

• Why is RAG usually better than fine-tuning for company knowledge?
• What does LoRA train instead of all weights?
• Why combine vector and keyword search?