Fine-tuning vs RAG: The 'Teaching vs. Memorizing' Mental Model

“Our AI keeps responding in English instead of Vietnamese. Should we fine-tune?”

“Our AI doesn’t know about our new product launched last week. Should we fine-tune?”

The answer to both is almost never the same. Yet engineers confuse the two constantly — and one wrong choice costs weeks and thousands of dollars.

---

Part 1: Foundations (The Mental Model)

The Medical School Analogy

RAG = Giving a doctor a reference book before every patient visit.

• “Here’s relevant information for this patient. Now diagnose.”
• The doctor’s underlying medical knowledge is unchanged.
• Perfect for: current information, company-specific data.

Fine-tuning = Sending the doctor to actual medical school.

• The doctor’s brain is re-trained. They internalize new knowledge and behaviors.
• Perfect for: changing how the model behaves, speaks, and reasons.

                    RAG                         Fine-tuning
Use when:    "Model lacks knowledge"      "Model lacks skill/style"
Cost:        Low (just indexing)          High ($$ GPU hours)
Updatable:   Instantly (re-index)         Hard (retrain)
Example:     "Know our FAQ"               "Always respond in our brand voice"

---

Part 2: The Investigation (When Fine-Tuning Wins)

Fine-tuning changes the model’s weight — its fundamental behavior. Use it when you need:

• Consistent format/style: “Always respond as bullet points in markdown.”
• Domain language: Medical jargon, legal language, code in a specific style.
• Task specialization: A model that only does SQL generation, fast and reliably.
• Language/dialect: Teaching a model to write natural Vietnamese (not translated-sounding).

When RAG is enough (use this first, always):

• The model just needs up-to-date facts it doesn’t know.
• You need to cite sources in your answer.
• Data changes frequently (product catalog, pricing).

---

Part 3: The Diagnosis (LoRA — Fine-tuning Without a Supercomputer)

Full fine-tuning updates ALL of a model’s billions of parameters. Prohibitively expensive.

LoRA (Low-Rank Adaptation) is the breakthrough that made fine-tuning accessible. Instead of updating all weights, it adds small adapter matrices to key layers. Only the adapters are trained (~1% of parameters).

Full Fine-Tuning: Update 7 billion parameters → needs 80GB GPU × 4 days
LoRA Fine-Tuning: Update ~70 million adapter params → needs 16GB GPU × 2 hours

Fine-tuning with Unsloth + LoRA (Python)

from unsloth import FastLanguageModel
from trl import SFTTrainer
from datasets import Dataset

# Load base model with 4-bit quantization (fits on a single consumer GPU)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b",
    max_seq_length=2048,
    load_in_4bit=True,  # 4-bit quantization: 8B model fits in ~6GB VRAM
)

# Apply LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,             # LoRA rank: higher = more capacity but more params
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],  # Which layers to adapt
)

# Your training data (instruction → response pairs)
data = Dataset.from_list([
    {"text": f"### Instruction:\n{ex['input']}\n\n### Response:\n{ex['output']}"}
    for ex in your_training_data
])

trainer = SFTTrainer(
    model=model,
    train_dataset=data,
    dataset_text_field="text",
    max_seq_length=2048,
)
trainer.train()

# Save only the adapters (small: ~50MB vs 16GB for the full model)
model.save_pretrained("my-lora-adapter")

---

Part 4: The Resolution (Decision Framework)

Problem: "AI doesn't know X"
    │
    ├── X changes frequently? → RAG (re-index = done)
    │
    ├── X is private/proprietary docs? → RAG
    │
    └── X is a skill/behavior/style? → Fine-tune
         │
         ├── Budget < $100? → LoRA on open model (Llama 3, Mistral)
         │
         └── Budget flexible? → OpenAI fine-tuning API (pay per token)

OpenAI Fine-Tuning API (Managed, No GPU)

from openai import OpenAI
import json

client = OpenAI()

# 1. Upload training data (JSONL format, min 10 examples)
with open("training.jsonl", "w") as f:
    for ex in training_data:
        f.write(json.dumps({
            "messages": [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": ex["question"]},
                {"role": "assistant", "content": ex["answer"]}
            ]
        }) + "\n")

file = client.files.create(file=open("training.jsonl", "rb"), purpose="fine-tune")

# 2. Start fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-4o-mini"  # Fine-tune the smaller model (cheaper)
)

# 3. Use your fine-tuned model
response = client.chat.completions.create(
    model=job.fine_tuned_model,  # e.g., "ft:gpt-4o-mini:acme:v1:abc123"
    messages=[{"role": "user", "content": "..."}]
)

---

Final Mental Model

RAG          → Give the doctor a reference book. Instant. Citable. Updatable.
Fine-tuning  → Send the doctor to med school. Permanent. Expensive. Powerful.

LoRA         → Surgically add adapter layers. Train 1% of params. 90% of the effect.
Full FT      → Retrain the entire brain. 100x more expensive. Rarely necessary.

Start with RAG. Fine-tune only when RAG can't fix it.

The 2026 rule: 90% of AI product problems are solved by better prompts + RAG. Fine-tune when you’ve exhausted both. LoRA when you fine-tune.