# Quantization in Practice: Compressing Models Yourself

[Skip to content](#lm-inhoud)Network/[NL](/en/quantisatie-in-de-praktijk-zelf-een-model-comprimeren)EN[Hubhub.llmnet.nlCompare models on task, language, cost and licence.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organisation, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fquantisatie-in-de-praktijk-zelf-een-model-comprimeren&text=Quantization%20in%20Practice%3A%20Compressing%20Models%20Yourself)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fquantisatie-in-de-praktijk-zelf-een-model-comprimeren)[](https://www.reddit.com/submit?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fquantisatie-in-de-praktijk-zelf-een-model-comprimeren&title=Quantization%20in%20Practice%3A%20Compressing%20Models%20Yourself)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fquantisatie-in-de-praktijk-zelf-een-model-comprimeren&text=Quantization%20in%20Practice%3A%20Compressing%20Models%20Yourself)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fquantisatie-in-de-praktijk-zelf-een-model-comprimeren)[](https://www.reddit.com/submit?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fquantisatie-in-de-praktijk-zelf-een-model-comprimeren&title=Quantization%20in%20Practice%3A%20Compressing%20Models%20Yourself)[](#)

 
# Quantization in practice: compressing a model yourself

 By Ivo Donker — compiled with AI assistance (Claude & Gemini) · August 15, 2026

 
 Minimum hardware test environment: Ubuntu 24.04 LTS, 32 GB DDR5 RAM, Nvidia RTX 4070 (12 GB VRAM), 100 GB free NVMe storage.

 Software versions used: llama.cpp release b3600 (August 2026), Python 3.11.9, GGUF specification v3.

 Target model in this guide: Qwen 2.5 7B / Llama 3.1 8B Instruct (original 16-bit FP16 safetensors).

 

 In the local AI workflow, this article sits between selecting a base model and actually running an optimized server: we move from choosing to the technical preparation and tuning of weights. Before you start quantizing, it is essential to verify that your workstation meets the basic requirements for video memory and system bus width; consult [what hardware you need for local LLMs](https://gids.llmnet.nl/en/hardware-voor-lokale-llm) to determine how much VRAM your setup has available. For a stable compilation environment with C++ tools, a Linux installation is recommended; read [running local LLMs on Linux](https://gids.llmnet.nl/en/lokale-llm-op-linux) to set up the required build-essentials and CUDA toolkits. If you first want to determine which architecture type best fits your intended task, you can navigate to [choosing a local model](https://gids.llmnet.nl/en/lokaal-model-kiezen) for a comparison between base and instruct models.

 Quantization reduces the precision of individual model parameters (weights) from 16-bit floating-point numbers to compact 4-, 5-, or 8-bit representations, allowing a model to take up to 70% less memory with minimal loss of reasoning power; for the theoretical mathematics and differences between data types, see [the guide to quantization and data types](https://gids.llmnet.nl/en/kwantisatie-uitgelegd). In this article, we execute the entire conversion process ourselves on the command line using llama.cpp.

 
## Why quantize a model yourself instead of downloading it?

 Platforms like Hugging Face offer thousands of ready-made GGUF files, often provided by community members. Still, there are compelling reasons to take control of the conversion and compression process yourself. New model architectures or recent checkpoints of fine-tuned models are often not immediately available as GGUF. As a developer, you don't want to wait for someone else to run a conversion script, especially when working with confidential domain-specific data or having just merged your own LoRA adapter with a base model.

 In addition, running your own quantization pipeline gives you full control over the selected matrix types (such as k-quants with varying precision per layer) and the use of a representative importance matrix (imatrix). This prevents a generic quantization from degrading critical attention heads. You also ensure that the source file hasn't been tampered with via modified tokenizers or corrupt metadata.

 
## The building blocks: compiling llama.cpp with hardware acceleration

 The core of the quantization toolchain is llama.cpp. To run this efficiently, we compile the C++ source code with support for your specific GPU architecture (CUDA for Nvidia, ROCm for AMD, or Metal for Apple Silicon). We clone the repository and build the binaries llama-quantize and llama-imatrix.

 # Installeer vereiste pakketten (Ubuntu / Debian)
sudo apt update && sudo apt install -y git build-essential cmake ccache python3-pip python3-venv

# Clone de officiële llama.cpp repository
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# Bouw met CUDA ondersteuning (voor Nvidia GPU's)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j $(nproc)

# Maak een geïsoleerde Python-omgeving aan voor conversiescripts
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

 Once the build process is complete, you will find the executables in the build/bin/directory. Verify whether the CUDA backend is active using the command ./build/bin/llama-cli --version . This ensures that subsequent matrix computations do not run exclusively on the CPU, which is substantially slower.

 
## Step 1: Downloading the source model (Safetensors format)

 Quantization always starts with full-precision weights (16-bit Float or BFloat16), typically stored in the safetensors format. Never download an already compressed model to quantize it again; cumulative rounding errors lead to irreversible quality loss (quantization artifacts).

 # Installeer de Hugging Face CLI tool
pip install huggingface_hub

# Download een 8B model in FP16 / BF16 safetensors formaat
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct \
 --local-dir ./models/Llama-3.1-8B-Instruct \
 --local-dir-use-symlinks False \
 --exclude "*.pth" "original/*"

 After downloading, verify that the file config.json, the tokenizer.json , and all .safetensors files are present in the target directory. Missing tokenizer files will cause the conversion script to fail when parsing special tokens.

 
## Step 2: Converting Safetensors to uncompressed GGUF (FP16 / BF16)

 Before reducing bits, the safetensors files must be converted to the GGUF format. This format consolidates all model tensors, vocabulary arrays, and hyperparameter metadata into a single file or a structured set of shards.

 # Voer het conversiescript uit vanuit de actieve venv
python3 convert_hf_to_gguf.py ./models/Llama-3.1-8B-Instruct \
 --outfile ./models/llama-3.1-8b-f16.gguf \
 --outtype f16

 For an 8B model, this step produces a file of approximately 15 to 16 GB. This file still retains the exact numerical precision of the source model. Keep in mind that you need at least 16 GB of free RAM during this step to reorder the tensor structures in system memory before writing them to disk.

 
## Step 3: Generating an Importance Matrix (imatrix)

 Standard quantization treats every matrix and layer using a uniform statistical scheme. However, certain weights in a neural network are disproportionately critical for logical coherence and syntax. By using an importance matrix to calculate (imatrix) based on a calibration dataset, measures llama.cpp which parameters are most sensitive to precision loss. Highly sensitive layers subsequently retain higher bit precision, while less critical tensors are compressed more aggressively.

 # Download een representatieve kalibratietekst (bijvoorbeeld WikiText of custom NL teksten)
curl -L -o ./models/calibration_data.txt https://raw.githubusercontent.com/ggerganov/llama.cpp/master/prompts/wiki.test.raw

# Bereken de imatrix via GPU-versnelling
./build/bin/llama-imatrix \
 -m ./models/llama-3.1-8b-f16.gguf \
 -f ./models/calibration_data.txt \
 -o ./models/llama-3.1-8b.imatrix \
 -ngl 99 \
 -c 512 \
 --chunks 64

 On a modern GPU, this process takes about two to five minutes for an 8B model. The resulting .imatrix file is only a few megabytes in size, but significantly improves the perplexity score of lower quantizations (such as Q4_K_M or Q3_K_M).

 
## Step 4: Performing the quantization to K-Quants

 Now that all preparations are in place, we start the actual quantization with llama-quantize. Here, we choose modern k-quant formats (such as Q4_K_M or Q5_K_M), where different weight blocks within the network are stored at different bit depths for an optimal balance between file size and performance.

 # Kwantisatie naar Q4_K_M met behulp van de gegenereerde imatrix
./build/bin/llama-quantize \
 --imatrix ./models/llama-3.1-8b.imatrix \
 ./models/llama-3.1-8b-f16.gguf \
 ./models/llama-3.1-8b-Q4_K_M.gguf \
 Q4_K_M

# Kwantisatie naar Q5_K_M voor een hogere redeneerprecisie
./build/bin/llama-quantize \
 --imatrix ./models/llama-3.1-8b.imatrix \
 ./models/llama-3.1-8b-f16.gguf \
 ./models/llama-3.1-8b-Q5_K_M.gguf \
 Q5_K_M

 The quantization step processes the tensors sequentially and takes an average of 30 to 90 seconds. Once completed, the file is llama-3.1-8b-Q4_K_M.gguf reduced to approximately 4.9 GB, which easily fits within an 8 GB or 12 GB GPU.

 
 
 
 
 Quantization Type | 
 Average Bit Depth | 
 Size (8B Model) | 
 Perplexity Increase (Lower = Better) | 
 Recommended Use Case | 
 

 
 
 
 FP16 (Base) | 
 16.0 bpw | 
 15.5 GB | 
 +0.00 (Reference) | 
 Archival, fine-tuning base | 
 

 
 Q8_0 | 
 8.5 bpw | 
 8.5 GB | 
 +0.004 | 
 Maximum precision, ample VRAM | 
 

 
 Q5_K_M | 
 5.5 bpw | 
 5.7 GB | 
 +0.035 | 
 Best balance for reasoning tasks | 
 

 
 Q4_K_M | 
 4.5 bpw | 
 4.9 GB | 
 +0.082 | 
 Standard for 8 GB / 12 GB GPUs | 
 

 
 Q3_K_M | 
 3.4 bpw | 
 3.7 GB | 
 +0.245 | 
 Tight VRAM budgets, noticeable quality loss | 
 

 
 IQ2_XXS | 
 2.0 bpw | 
 2.6 GB | 
 +0.810 | 
 Edge devices, severe degradation of logic | 
 

 
 
 

 
## Step 5: Quality verification and measuring Perplexity

 Completing the compression without error messages does not automatically mean that the output content has remained intact. To objectively determine how much precision has been lost, we calculate the perplexity of the quantized model compared to the FP16 original. Perplexity is a mathematical metric measuring how surprised the language model is by a standard set of texts.

 # Test de perplexity van het gekwantiseerde bestand
./build/bin/llama-perplexity \
 -m ./models/llama-3.1-8b-Q4_K_M.gguf \
 -f ./models/calibration_data.txt \
 -c 512 \
 -ngl 99

 A slight increase in perplexity (for example from 5.42 to 5.50) is completely normal with 4-bit quantization and has barely any noticeable impact on text quality in everyday applications. However, if the perplexity spikes to values above 7.0, severe degradation has occurred due to incorrect calibration or damaged layers.

 
## Practical test: Quality difference on Dutch instructions

 Dutch syntax and grammar are more sensitive to aggressive quantization than English, as Dutch tokens generally have fewer training parameters allocated in multilingual base models. Anyone looking to work systematically on language optimization is well-advised to read the guidelines on how to [make AI perform better in Dutch](https://gids.llmnet.nl/en/beter-nederlands) to mitigate typical anglicisms and sentence structure issues.

 Let's present a test prompt to both the Q4_K_M and the Q5_K_M variant to verify how strictly instructions are followed:

 Prompt: "Leg in exact drie genummerde punten uit wat het verschil is tussen werkgeheugen en videogeheugen bij het draaien van een lokaal taalmodel."

Uitvoer Q5_K_M:
1. Werkgeheugen (RAM) bevindt zich op het moederbord en wordt aangesproken door de CPU; het is relatief traag maar vaak beschikbaar in grotere capaciteiten (bijv. 32 tot 64 GB).
2. Videogeheugen (VRAM) zit direct op de videokaart (GPU) en biedt een vele malen hogere bandbreedte (tot wel 1 TB/s), wat essentieel is voor snelle token-generatie.
3. Als een model volledig in het VRAM past draait het op maximale snelheid; zodra lagen moeten uitwijken naar het reguliere RAM (offloading) daalt de verwerkingssnelheid drastisch.

Uitvoer Q3_K_S (ter vergelijking bij te sterke compressie):
1. RAM is voor normale computerdingen en CPU.
2. VRAM is van de videokaart en is sneller voor AI tensors.
3. Als geheugen vol is stopt het of gaat langzaam. (Punt 3 mist detail en nuance).

 In practice, we observe that Q4_K_M and Q5_K_M flawlessly maintain the grammatical structure and the strict instruction constraint ("exactly three numbered points"), while Q3_K_S begins to compromise on depth and vocabulary choice.

 
## Integration into Ollama via a custom Modelfile

 Once the GGUF file is validated, you can easily integrate it into your daily workflow, such as with Ollama. To do this, we write a simple Modelfile pointing to the local file path.

 # Maak een bestand genaamd Modelfile aan
FROM ./models/llama-3.1-8b-Q4_K_M.gguf

# Stel parameters in voor deterministisch gedrag
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER stop ""
PARAMETER stop ""

# Definieer het chatsjabloon van Llama 3.1
TEMPLATE """{{ if .System }}system

{{ .System }}{{ end }}{{ if .Prompt }}user

{{ .Prompt }}{{ end }}assistant

{{ .Response }}"""

SYSTEM """Je bent een behulpzame, feitelijke assistent die altijd antwoordt in correct Nederlands."""

 Next, register the model in Ollama using the build command:

 # Bouw het model lokaal in Ollama
ollama create llama3-custom-q4 -f Modelfile

# Start een interactieve sessie
ollama run llama3-custom-q4

 
## Privacy, isolation, and data security

 Quantizing and running language models locally guarantees that not a single byte of input data, document text, or internal weights is sent to external cloud servers. All operations — from compiling the source code to generating the imatrix and executing inferences — take place fully isolated within your local file system. Organizations that enforce strict confidentiality or must comply with GDPR can run this process on an air-gapped workstation without an active internet connection once the source code and weights have been fetched; for structural assurance, consult the overview on [privacy-friendly AI use](https://gids.llmnet.nl/en/privacyvriendelijk-ai) to definitively eliminate data leaks during model invocations.

 
## Assessing practical risks and reliability

 Although quantization saves compute power, it introduces a subtle risk: small rounding errors in the weights can cause a model to hallucinate more frequently when handling factual questions or mathematical calculations. Therefore, always critically evaluate the responses of compressed models; consult the guide on [fact-checking AI responses](https://gids.llmnet.nl/en/ai-antwoorden-factchecken) to systematically apply source verification and checks to generated texts. When setting up local AI for shared use within a household or on a home server, it is wise to review the guide on [Using AI safely at home](https://gids.llmnet.nl/en/ai-veilig-thuis) to ensure adequate protection of sensitive data and child-friendly filtering.

 For developers looking to deploy quantization as part of autonomous systems, pipelines, and tool-calling agents, the specialized guide on [Becoming an AI agent engineer in 2026](https://leren.llmnet.nl/en/ai-agent-engineer-worden-2026) provides a comprehensive overview of the skills required to reliably connect local language models to external APIs and data sources.

 
## Common errors and troubleshooting

 Specific errors may occur during the quantization process. Below are the most common bottlenecks and their corresponding fixes:

 
 
- CUDA out of memory during imatrix calculation: Reduce the context length in the imatrix command using the flag -c 256 or reduce the number of GPU layers via -ngl 20 so that part of the computation runs in system RAM.
 
- KeyError in conversion script: This usually indicates an unconventional model architecture that is not yet supported by the current release of llama.cpp. Run git pull in the llama.cpp repository and recompile the binary.
 
- Garbage output (gibberish) after quantization: This is almost always caused by a tokenizer mismatch or missing special tokens in the source directory. Ensure that tokenizer.json and tokenizer_config.json are present when running convert_hf_to_gguf.py.
 

 With the right combination of convert_hf_to_gguf.py, llama-imatrix and llama-quantize your workstation has a full-fledged pipeline to tailor any open model precisely to your available VRAM, while maintaining sharp reasoning performance and complete privacy.