Running Large Language Models (LLMs) locally offers unprecedented control, privacy, and speed. However, in their original form, large open-source models like Llama 3 or Mistral require massive amounts of system memory (RAM) and video memory (VRAM). This is where quantization comes in: a technique that allows you to compress these models significantly so they run seamlessly on consumer hardware.
What exactly is quantization?
At its core, a neural network stores weights (parameters) as high-precision numbers, typically as 16-bit floating-point numbers (FP16) or even 32-bit (FP32). This means each parameter costs 2 to 4 bytes of memory. Consequently, a model with 70 billion parameters quickly requires more than 140 GB of memory.
With quantization, these continuous precision values are converted into discrete, smaller numbers, such as 8-bit, 5-bit, or even 4-bit memory representations (for example, INT4). This drastically reduces the memory footprint. A 16-bit model that is originally 30 GB in size fits into less than 8 GB after 4-bit quantization.
Indication: By quantizing to a 4-bit format, you reduce memory consumption to roughly a quarter of the original, while the model in practice retains 95% or more of its original performance.
The GGUF format and modern standards
Within the local LLM world, GGUF (developed by the community around llama.cpp) has become the absolute standard for quantized models. GGUF replaced the older GGML format and offers better support for metadata, tokenizers, and hybrid execution across both CPU and GPU.
Within GGUF, a distinction is made between various quantization levels, which are designated by standardized labels:
| GGUF Variant | Avg. Bits per Weight | Quality Retention (Indication) | Suitable for |
|---|---|---|---|
Q8_0 |
8.5 bits | Virtually identical to original (FP16) | Systems with ample VRAM/RAM |
Q6_K |
6.6 bits | Excellent, negligible loss | Optimal balance when space is abundant |
Q4_K_M |
4.8 bits | Good to very good (gold standard) | Most consumer PCs and laptops |
Q3_K_S |
3.5 bits | Noticeable quality loss | Only if the model does not fit otherwise |
Quality versus Memory: The practice
It is a misconception that a lower number of bits per weight directly leads to unusable responses. Thanks to advanced modern quantization techniques (such as k-quantization), the most important weights are preserved with higher precision, while less critical parameters are compressed more heavily.
Still, there is a limit. From 3 bits and lower, you typically see models struggle with complex reasoning tasks, coding, or maintaining context. For general use and chat applications, the Q4_K_M or Q5_K_M variant is almost always the smartest choice.
How to choose the right variant for your hardware?
To determine which GGUF variant you can download and run, follow this simple step-by-step guide:
- Assess your hardware: Check how much VRAM your graphics card (GPU) has. What does not fit in the GPU will run on the regular system memory (RAM) via the CPU, which is slower.
- Calculate the memory margin: Add about
20%extra space to the size of the quantized model (in GB) for the KV cache (context memory during inference). - Choose your variant: Do you have just enough VRAM for a Q4 variant? Then feel free to choose
Q4_K_M. Can you load the model entirely into memory with room to spare? Then upgrade toQ5_K_MorQ8_0for maximum accuracy.
By making an informed choice, you get the most out of your local AI setup without having to immediately purchase expensive enterprise hardware.
Read more about setting up your system in our guide on hardware choices for local AI.