Fine-Tuning Open Models

Fine-tuning sounds like the answer to every customization problem. In practice, it's one tool in a larger toolkit — and often not the first one you should reach for.

When fine-tuning makes sense

Fine-tune when you need the model to consistently follow a format, adopt a tone, or perform a narrow task that prompting and RAG can't reliably deliver.

Skip fine-tuning when:

• Your knowledge changes frequently (use RAG instead)
• You need factual recall from documents (use RAG)
• A few-shot prompt already works 95% of the time

Data preparation

Quality beats quantity. A few hundred well-curated examples outperform thousands of noisy ones.

Each training example should include:

• A clear instruction
• Optional context
• The ideal output

Remove duplicates, fix formatting inconsistencies, and hold out 10–15% for evaluation.

LoRA vs full fine-tune

LoRA (Low-Rank Adaptation) trains small adapter matrices while freezing base weights. It's faster, cheaper, and you can swap adapters per task.

Full fine-tune updates all parameters. Use it only when LoRA plateaus and you have the compute budget.

# Typical LoRA config (conceptual)
lora_config = {
    "r": 16,
    "lora_alpha": 32,
    "target_modules": ["q_proj", "v_proj"],
    "lora_dropout": 0.05,
}

Evaluation

Don't trust loss curves alone. Build an eval set with:

• Happy-path examples
• Edge cases and adversarial inputs
• Regression tests from production failures

Compare against your baseline (prompt-only or RAG) with the same eval set.

Deployment

Package the base model + LoRA adapter. Version both. Monitor drift in production and retrain when eval scores drop.

Bottom line

Start with prompting. Add RAG for knowledge. Fine-tune for behavior and format. Small teams can ship meaningful custom models with LoRA on a single GPU in an afternoon — if the data is clean.