Fine-Tuning Large Language Models: A Practical Guide
When fine-tuning is worth it, how to prepare chat-format data, and a minimal QLoRA training script with transformers, peft and trl, plus how to evaluate and merge the adapter.

We had a ticket-triage step that needed a small model to return one fixed JSON shape. With a long prompt containing a dozen examples, the base model got the format right most of the time, but every request carried those examples again, and the occasional malformed answer still broke a downstream parser. The question was whether to keep tuning the prompt or to teach the model the format once. That is the practical reason to fine-tune: not to make a model "smarter", but to make a specific behaviour cheap and reliable.
This guide covers the decision, the data, a minimal QLoRA script, and the evaluation that keeps you honest.
Quick takeaways
- Try prompting first, then retrieval. Fine-tune for style, format and narrow behaviour, not for fresh facts.
- Data quality beats quantity: a few hundred clean, consistent examples often move a narrow task more than thousands of noisy ones.
- LoRA trains small adapter matrices; QLoRA does the same on a 4-bit base model, which fits on a single consumer GPU for small models.
- Always compare against the base model with a good prompt on the same held-out set. If the gap is small, do not ship the fine-tune.
- Check for forgetting: keep a few general prompts and re-run them after training.
Should I fine-tune, use RAG, or just improve the prompt?
I use this order, and stop at the first step that works:
1. Prompting -> output is wrong because instructions are unclear
2. Retrieval (RAG) -> output is wrong because the model lacks your facts
3. Fine-tuning -> output is right in content but wrong in style/format,
or the prompt is too long/expensive to repeat every call
Facts that change (prices, policies, documentation) belong in retrieval, because a fine-tuned model cannot be updated without retraining. Behaviour that must be consistent (a JSON schema, a tone of voice, a classification taxonomy) belongs in the weights. For the first two steps, see Prompt Engineering Patterns and RAG chunking strategies.
What is the difference between full fine-tuning, LoRA and QLoRA?
Full fine-tuning updates every weight. It gives the most flexibility, needs the most memory, and risks erasing general abilities (catastrophic forgetting). LoRA freezes the base model and trains small low-rank matrices inserted into the attention and MLP layers, so the trainable parameters are a tiny fraction and the result is a small adapter file you can swap per task. QLoRA keeps the frozen base model in 4-bit precision during training, which is what lets a 7B-parameter model train on one 24 GB GPU. For most application work, LoRA or QLoRA is the right default.
How should I prepare the training data?
Use the chat format your model was trained with, one JSON object per line. Keep the system prompt short and identical across examples; the point of fine-tuning is that you no longer need the long prompt.
{"messages": [{"role": "system", "content": "Classify the support ticket. Reply with JSON only."}, {"role": "user", "content": "The invoice PDF shows the wrong VAT rate for Turkey."}, {"role": "assistant", "content": "{\"category\": \"billing\", \"priority\": \"medium\", \"language\": \"en\"}"}]}
Review examples by hand. Remove duplicates, fix inconsistent labels, and make sure the answers are exactly what you would accept in production. Split the data before training and never look at the validation set while tuning. Use 10 percent for validation, or a fixed set that stays the same across experiments.
What does a minimal QLoRA training script look like?
This uses transformers, peft and trl. Library APIs move quickly, so check argument names against the versions you install (for example, newer trl releases take the tokenizer as processing_class):
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer
BASE = "Qwen/Qwen2.5-7B-Instruct" # any instruct model whose license fits your use
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto")
data = load_dataset("json", data_files="tickets.jsonl")["train"].train_test_split(test_size=0.1, seed=42)
lora = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules="all-linear",
task_type="CAUSAL_LM",
)
args = SFTConfig(
output_dir="out/ticket-triage",
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
logging_steps=10,
eval_strategy="epoch",
save_strategy="epoch",
bf16=True,
)
trainer = SFTTrainer(
model=model,
args=args,
train_dataset=data["train"],
eval_dataset=data["test"],
processing_class=tokenizer,
peft_config=lora,
)
trainer.train()
trainer.save_model("out/ticket-triage/adapter")
The conversational messages format is converted with the model's chat template by SFTTrainer. Typical starting points for LoRA are a learning rate between 1e-4 and 3e-4, two to four epochs on a small dataset, and the largest batch that fits using gradient accumulation. Watch the validation loss: if it starts rising while the training loss keeps falling, you are overfitting. Stop earlier, add data, or lower the rank.
How do I know the fine-tune is actually better?
Loss tells you very little about usefulness. Evaluate on the held-out set with task-specific checks, the same ones you would use in production:
import json
def score(predictions: list[str], gold: list[dict]) -> dict:
valid = exact = 0
for raw, expected in zip(predictions, gold):
try:
parsed = json.loads(raw)
except json.JSONDecodeError:
continue
if not isinstance(parsed, dict):
continue
valid += 1
exact += parsed.get("category") == expected["category"]
n = len(gold)
return {"valid_json": valid / n, "category_accuracy": exact / n}
Run it for three candidates on identical inputs: the base model with your best prompt, the base model with the short prompt, and the fine-tuned model with the short prompt. Then run twenty or so general prompts (summaries, simple questions, refusals) and read the answers to catch forgetting. Building a proper test harness is covered in Building Evaluation Frameworks for LLM Applications.
How do I deploy the adapter?
You can load the adapter on top of the base model at inference time, which lets one base model serve several tasks. Or merge it into the weights for simpler serving. Merge into a base loaded in 16-bit, not the 4-bit one:
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="auto", device_map="auto")
merged = PeftModel.from_pretrained(base, "out/ticket-triage/adapter").merge_and_unload()
merged.save_pretrained("out/ticket-triage/merged")
For packaging and speed on edge hardware, see High-Performance Edge Inference.
What should I do this week?
Collect 200 real inputs for the narrow task you have in mind, label them carefully, and measure your current prompt on them. If the base model already passes, you have saved yourself a project. If it fails on format or consistency, you have your training set and your baseline in one step.
Uğur Kaval
AI/ML Engineer & Full Stack Developer specializing in building innovative solutions with modern technologies. Passionate about automation, machine learning, and web development.


