Fine-tuning locally with LoRA: a practical guide
Locally adapting large language models is an effective method for tailoring an AI model to a specific task, writing style, or output format. With techniques such as Low-Rank Adaptation (LoRA), it's no longer necessary to retrain billions of parameters. Instead, you add a small, attachable layer to the base model. In this guide, we walk through the entire process of local fine-tuning with LoRA: from determining the right strategy to data preparation, settings, evaluation, and integration.
When Is Fine-Tuning the Right Choice?
Before you start collecting data and setting up a training environment, it's essential to determine whether fine-tuning is actually the right solution for your problem. A common misconception is that fine-tuning serves to teach a model new knowledge. In practice, however, a language model struggles to reliably store exact facts through weight adjustments. For adding current, specific, or dynamic information, a Retrieval-Augmented Generation (RAG) system is almost always the better route.
Fine-tuning primarily lends itself to adapting the form, style, and behavior of the output. Think of enforcing a specific JSON schema, adopting a particular company tone, learning a fixed domain language, or consistently following complex instructions. For a comprehensive comparison between these approaches, consult the overview of fine-tuning vs. prompting vs. RAG on the knowledge network.
Core rule: Use RAG if the model needs to look up factual information; use fine-tuning if the model needs to phrase, structure, or process the information differently.
What Can You Realistically Achieve Locally?
Training a large language model from scratch (pre-training) requires enormous amounts of compute power and data centers full of specialized hardware. At home or on a local workstation, you therefore never train the full base model. With LoRA, you freeze the base model's original weights and attach small, adjustable weight matrices to specific layers. This drastically reduces the number of trainable parameters, which greatly lowers memory requirements and processing load.
To understand how these adapters work at a mathematical level without permanently overwriting the base model's parameters, you can consult the article on LoRA and adapters explained . In practice, this means you can successfully train an adapter on existing open-weights models using a consumer GPU with sufficient memory. Do make sure the physical hardware is balanced; check the requirements for hardware for local LLMs.
The Dataset: The Foundation of the Project
The quality and structure of the dataset largely determine the ultimate success of the LoRA adapter. A small, high-quality file consistently delivers better results than a large set full of inconsistencies.
Size of the Training Set
For learning a specific writing style or a fixed output format, a few hundred to a few thousand quality examples are often enough. What matters is that the dataset shows enough variation within the desired structure, without the model constantly repeating the same sentences.
Structuring in the Correct Conversation Format
The training examples must have exactly the same format as the interaction the base model expects. If the model works with specific roles (such as system, user, and assistant), each training example must be structured according to that exact format. The input represents the user's prompt, and the desired output is the assistant's response on which the model adjusts its weights.
Creating a Holdout Set
Split the collected data from the start into two parts: a training set and a held-out validation set. The validation set is never used during the gradient updates. This part serves solely to check, during and after training, whether the model is truly generalizing the patterns or simply memorizing the training sentences.
Data Quality and the Impact of Noise
Language models are extremely sensitive to patterns in the training data. When a dataset contains spelling errors, contradictory answers to similar questions, or inconsistent formatting, the adapter will directly adopt this unpredictable behavior.
| Problem in Dataset | Consequences for the Adapter | Solution in Advance |
|---|---|---|
| Inconsistent JSON keys | Model generates invalid or inconsistent structures | Validate the JSON structure of each example with a parser |
| Contradictory answers | Model gets confused and produces unstable outcomes | Remove or harmonize duplicate/overlapping questions |
| Inconsistent language use | Style and tone become unpredictable | Set clear guidelines for the assistant responses |
| Redundant system headers | Model learns incorrect instruction markers | Clean up all raw formatting characters before processing |
Thoroughly clean up the dataset before starting training. Use automated scripts to check for empty fields, deviating lengths, and incorrect formatting. Also perform manual spot checks to verify that the answers exactly match the desired standard.
The Most Important Settings in Plain Language
When setting up training, you'll encounter a number of hyperparameters. It's important to understand how these dials affect training, rather than simply copying random values.
Adapter Rank (Rank / r)
The rank determines the size of the adapter matrices. A lower rank means fewer parameters are added, which costs less memory and simplifies training. A higher rank gives the adapter more capacity to store complex patterns but increases the risk of overfitting and demands more from the hardware.
Target Layers (Target Modules)
You can choose to attach the LoRA adapter only to the attention layers or also to the feed-forward networks within the model. Applying LoRA to all linear layers gives the adapter the most flexibility to adjust the style, but increases the number of trainable parameters.
Learning Rate
The learning rate determines how strongly the weights are adjusted at each step based on the error margin. A learning rate that's too high can cause the model to unlearn existing knowledge or make the training process unstable. A learning rate that's too low results in the adapter barely changing and failing to adopt the desired style.
Number of Epochs and Batch Size
The number of epochs indicates how often the model sees the full training set. The batch size determines how many examples are processed together for one weight update. When hardware memory is limited, a small micro-batch size can be combined with gradient accumulation to still achieve a stable virtual batch size.
The Prompt Template: The Most Common Pitfall
Large language models are trained with specific delimiter tokens to indicate when a system instruction, user question, or assistant response begins and ends. Strictly adhering to this exact prompt template is crucial during fine-tuning.
If you deviate from the base model's expected tokens during training — for example by inventing your own delimiters or mislabeling roles — the model learns the relationship between input and output incorrectly. In that case, the adapter will behave strangely locally or appear to do nothing at all when you address it via a standard runner. Make sure, therefore, that when selecting the base model you know exactly which templates are used. Consult the article on downloading and managing models for more information on checking model cards and architecture specifications.
Recognizing and Preventing Overfitting
Overfitting occurs when the adapter memorizes the training examples word for word instead of understanding the underlying structure. As a result, the model loses its general language ability and can no longer formulate useful answers outside the exact training sentences.
You recognize overfitting by monitoring the loss value during training for both the training set and the held-out validation set:
- Healthy training: Both the training loss and the validation loss steadily decrease.
- Overfitting: The training loss keeps decreasing, but the validation loss starts to rise. This is the signal that the adapter is becoming too specific and training should be stopped (early stopping).
Merging or Loading the Adapter Separately
Once the training process is complete, you're left with a set of adapter weights. There are two ways to use this adapter in your local environment:
1. Loading Separately at Runtime
Some local runners can load the unmodified base model into memory and dynamically layer the LoRA adapter on top of it at startup. This has the advantage that the base model stays clean and you can easily switch between different adapters without duplicating the full model files.
2. Merging
You can also permanently recompute the adapter's weights and combine them with the base model. This creates a new, standalone model file. You can then quantize this file and convert it to a format suitable for local distribution. How to subsequently set up a merged model within a local infrastructure can be found in the guide on customizing the Ollama Modelfile.
Testing Against the Base Model
After delivering the adapter, a systematic evaluation is necessary. Test the new adapter on the same questions as the unmodified base model to clearly map out the difference in output.
Within the Training Domain
Check whether the adapter correctly handles questions within the intended domain according to the desired format, the right style, and the requested parameters. Make sure the answers aren't simply copies of the training data.
Outside the Training Domain (Catastrophic Forgetting)
Also pose the model general questions unrelated to your training set. If an adapter has been trained too aggressively, the model may have partly lost its general reasoning ability or logical skills. This phenomenon is known as catastrophic forgetting. To approach evaluations in a structured way, you can use the step-by-step plan for setting up your own benchmark on the benchmark platform.
Storage, Version Control, and Reproducibility
A fine-tuning project is only complete once the entire pipeline is reproducible. Keeping only the final adapter file is insufficient if you want to make updates later or repeat the process on a new base model.
Therefore, always store the following components together and under version control:
- The exact dataset: Including the cleaning scripts used and the exact split between training and validation.
- The configuration file: All settings such as the rank, learning rate, target layers, and number of epochs.
- Base model identification: The exact version, hash, or commit ID of the base model used. After all, a small update to the base model can render an adapter unusable.
- The prompt template: The exact textual structure and special tokens applied during training.
By carefully preserving this information, you ensure your local fine-tuning projects remain manageable, transferable, and future-proof.
