# Quantization explained: running large models on small

[Skip to content](#lm-inhoud)Network/[NL](/en/kwantisatie-uitgelegd)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkwantisatie-uitgelegd&text=Quantization%20explained%3A%20running%20large%20models%20on%20small)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkwantisatie-uitgelegd)[](https://www.reddit.com/submit?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkwantisatie-uitgelegd&title=Quantization%20explained%3A%20running%20large%20models%20on%20small)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkwantisatie-uitgelegd&text=Quantization%20explained%3A%20running%20large%20models%20on%20small)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkwantisatie-uitgelegd)[](https://www.reddit.com/submit?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkwantisatie-uitgelegd&title=Quantization%20explained%3A%20running%20large%20models%20on%20small)[](#)By Ivo Donker — created with AI assistance (Claude & Gemini) · Last updated: July 27, 2026

# Quantization explained: running large models on small hardware

Everything about shrinking Large Language Models without losing valuable logic and context.

Running Large Language Models (LLMs) locally offers unprecedented control, privacy, and speed. However, in their original form, large open-source models like Llama 3 or Mistral require massive amounts of system memory (RAM) and video memory (VRAM). This is where quantization comes in: a technique that allows you to compress these models significantly so they run seamlessly on consumer hardware.

## What exactly is quantization?

At its core, a neural network stores weights (parameters) as high-precision numbers, typically as 16-bit floating-point numbers (FP16) or even 32-bit (FP32). This means each parameter costs 2 to 4 bytes of memory. Consequently, a model with 70 billion parameters quickly requires more than 140 GB of memory.

With quantization, these continuous precision values are converted into discrete, smaller numbers, such as 8-bit, 5-bit, or even 4-bit memory representations (for example, INT4). This drastically reduces the memory footprint. A 16-bit model that is originally 30 GB in size fits into less than 8 GB after 4-bit quantization.

Indication: By quantizing to a 4-bit format, you reduce memory consumption to roughly a quarter of the original, while the model in practice retains 95% or more of its original performance.

## The GGUF format and modern standards

Within the local LLM world, GGUF (developed by the community around llama.cpp) has become the absolute standard for quantized models. GGUF replaced the older GGML format and offers better support for metadata, tokenizers, and hybrid execution across both CPU and GPU.

Within GGUF, a distinction is made between various quantization levels, which are designated by standardized labels:

GGUF Variant | 
Avg. Bits per Weight | 
Quality Retention (Indication) | 
Suitable for | 

Q8_0 | 
8.5 bits | 
Virtually identical to original (FP16) | 
Systems with ample VRAM/RAM | 

Q6_K | 
6.6 bits | 
Excellent, negligible loss | 
Optimal balance when space is abundant | 

Q4_K_M | 
4.8 bits | 
Good to very good (gold standard) | 
Most consumer PCs and laptops | 

Q3_K_S | 
3.5 bits | 
Noticeable quality loss | 
Only if the model does not fit otherwise | 

## Quality versus Memory: The practice

It is a misconception that a lower number of bits per weight directly leads to unusable responses. Thanks to advanced modern quantization techniques (such as k-quantization), the most important weights are preserved with higher precision, while less critical parameters are compressed more heavily.

Still, there is a limit. From 3 bits and lower, you typically see models struggle with complex reasoning tasks, coding, or maintaining context. For general use and chat applications, the Q4_K_M or Q5_K_M variant is almost always the smartest choice.

## How to choose the right variant for your hardware?

To determine which GGUF variant you can download and run, follow this simple step-by-step guide:

- Assess your hardware: Check how much VRAM your graphics card (GPU) has. What does not fit in the GPU will run on the regular system memory (RAM) via the CPU, which is slower.

- Calculate the memory margin: Add about 20% extra space to the size of the quantized model (in GB) for the KV cache (context memory during inference).

- Choose your variant: Do you have just enough VRAM for a Q4 variant? Then feel free to choose Q4_K_M. Can you load the model entirely into memory with room to spare? Then upgrade to Q5_K_M or Q8_0 for maximum accuracy.

By making an informed choice, you get the most out of your local AI setup without having to immediately purchase expensive enterprise hardware.

Read more about setting up your system in our guide on [hardware choices for local AI](https://leren.llmnet.nl/en/).

© 2026 llmnet.nl — Local LLM Guide. All rights reserved.