Running large language models (LLMs) locally is often associated with expensive workstations, powerful graphics cards, and tens of gigabytes of dedicated VRAM. However, if you have an older laptop, a traditional office PC without a discrete GPU, or an older Intel-based Mac, you don't necessarily have to sit on the sidelines. With the right settings, a realistic model size, and targeted optimizations, it is entirely possible to squeeze usable performance out of existing hardware.
This guide is written as a practical handbook for optimizing your current system. We don't focus on buying new components, but on smartly configuring the resources you already have. For a general overview of system requirements for new systems, please consult our guide on hardware for local LLMs.
1. Realistically determining what your machine can handle
Before you start downloading model files, it's essential to understand which components of your computer determine performance when running a local language model.
System memory (RAM) as a hard upper limit
When running an LLM on a computer without a dedicated graphics card, the model is loaded entirely into regular system memory (RAM). Available RAM simply determines which model will or won't fit on your system. If a model file is larger than the available memory, the operating system will try to write data to the hard drive or SSD (swapping), resulting in an impractically slow interface.
As a rule of thumb, you should keep at least 2 to 3 gigabytes of RAM free for the operating system and background tasks. On a system with 8 GB of RAM, you effectively have about 5 GB of space for the AI software and the model.
CPU versus GPU and the role of memory bandwidth
Many people assume that the clock speed or the number of processor (CPU) cores is the main limiting factor. However, when an LLM generates text, the real bottleneck is almost always memory bandwidth. For each individual word (token) generated, a language model must stream the entire dataset of all weights from memory to the processor.
The memory bottleneck in practice: Older DDR3 or DDR4 memory in traditional PCs moves data at typically 20 to 40 gigabytes per second. By comparison, modern GPUs or Apple Silicon chips achieve bandwidths of 150 to over 800 GB/s. On older hardware, processing speed is therefore primarily a matter of the physical speed at which the RAM chips can send data to the CPU.
2. Choosing a smaller model instead of forcing a large one
The most effective way to get usable performance on limited hardware is to radically reduce the model size. A common mistake is trying to run a model with 7 or 13 billion parameters on a machine with 8 GB of RAM.
Instead, choose compact models specifically designed to perform efficiently at a lower parameter count. In the 0.5B, 1.5B, and 3B parameter categories, enormous quality improvements have been made in recent years. For more information on selecting suitable categories, see our guide on choosing a local model.
| Available RAM | Recommended parameter count | Indicative memory footprint for model |
|---|---|---|
| 4 GB RAM | 0.5B – 1.5B parameters | 1.5 to 2.5 GB |
| 8 GB RAM | 1.5B – 3B parameters | 2.5 to 5.0 GB |
| 16 GB RAM (older CPU) | 3B – 7B parameters (heavily quantized) | 5.0 to 9.0 GB |
3. Quantization as the main lever
Quantization is the technique of reducing the numerical precision of the weights in the AI model (for example, from 16-bit floating point to 4-bit or 2-bit integers). This drastically reduces the model file size and decreases the amount of data that must be sent over the memory bus per generated token.
For older hardware, quantization is not an option but a requirement. For a detailed explanation of the technical background and loss values, see our guide on quantization explained.
What you sacrifice at low precision
- Q4_K_M (4-bit): The gold standard for balancing quality and memory usage. Offers barely noticeable quality loss compared to the original model.
- Q3_K_M (3-bit): Provides extra room on constrained systems. Slight degradation occurs in logical reasoning and following complex instructions.
- Q2_K (2-bit): Results in an extremely small memory footprint, but the model may make more errors, distort facts, or fall into repetition. Only use this if you otherwise can't run a model at all.
4. Reducing context length to save memory
When you start a model, not only the weight matrix occupies memory, but also the so-called Key-Value (KV) cache. This cache stores the information from the ongoing conversation and the entered text.
By default, many modern models support a context window of 8,192 to 32,768 tokens. However, on a system with limited RAM, a large context window causes a dangerous increase in memory usage. As the conversation grows longer, the KV cache can demand several gigabytes of additional memory, causing the computer to unexpectedly freeze.
Practical context setting
Explicitly limit the context length in your software (such as Ollama or LM Studio) to 2048 tokens or even 1024 tokens on very old machines. If you see that the software is set to 4096 via a configuration parameter such as num_ctx, halve this value. This immediately gives the CPU breathing room and prevents memory spikes during long prompts.
5. Recognizing and preventing swap and thermal throttling
Older laptop hardware faces two physical obstacles during prolonged AI tasks: virtual memory (swapping) and overheating (thermal throttling).
Recognizing swapping
When the computer has too little physical RAM, the operating system uses the hard drive or SSD as temporary storage. You can recognize swapping by the following symptoms:
- The fan kicks in, the hard drive indicator blinks continuously, and the mouse pointer responds erratically.
- Generating the first letters takes minutes.
- Processing speed collapses to less than 1 token per second.
Solution: Close the application and choose a smaller or more heavily quantized model. Always stay well below the machine's physical RAM limit.
Thermal throttling
Processors in older laptops are not designed to run at 100% load on all cores for minutes at a time. When the temperature rises too high, the processor will automatically reduce its clock speed to prevent damage. As a result, the model's response time becomes noticeably slower over time.
Ensure good ventilation by placing the laptop on a hard, flat surface and removing dust from the ventilation grilles. If necessary, limit the number of allocated CPU threads in the AI runner's settings to $N-1$ or $N-2$ (where $N$ is the total number of processor cores) so the system doesn't constantly run at maximum capacity.
6. Freeing up system resources and stopping background processes
On a powerful modern PC, an open web browser with thirty tabs makes little difference. On an old machine with 8 GB of RAM, however, a web browser can already consume 2 to 3 GB of memory.
- Close heavy applications: Fully close web browsers, communication apps, video players, and photo editing programs before starting the AI environment.
- Prevent automatically loaded models: Many local AI tools keep the model in memory after use for a set timeout (e.g., 5 minutes). If you're trying out multiple models, make sure the old model is explicitly unloaded from memory before starting a new one.
- Use lightweight operating systems: On older PCs, running a lightweight Linux distribution can free up 1 to 2 GB of additional memory compared to a heavy Windows installation.
Apple laptops with an older operating system require a specific approach; for this, see our guide on installing Ollama on macOS.
7. Adjusting expectations: what is and isn't realistic?
It's important to have clear expectations about what is achievable when running a local model on an older machine. Performance should always be weighed against the desire to process data fully privately.
Realistic and suitable tasks
- Short text restructuring: Rewriting emails, checking grammar, and improving phrasing.
- Concise summaries: Summarizing short articles or individual paragraphs.
- Simple questions and answers: Asking questions about specific topics that don't require the model to perform complex reasoning steps.
- Code assistance for short functions: Help with simple scripts or explaining a line of code.
Unrealistic tasks for older hardware
- Processing long documents or books: Because the context window must be kept small, a long document simply doesn't fit in memory.
- Agentic chains and complex RAG systems: Systems that perform dozens of intermediate searches and steps in succession take unacceptably long on slow hardware.
- Heavy programming tasks: Models of 14B or 33B parameters that excel at programming are too large for this category of devices.
To check how your specific hardware's performance compares to other configurations, you can consult the benchmark results in our external test environment at LLMnet Benchmark.
8. When running locally is no longer worthwhile
Although optimization can go a long way, every hardware generation has its limits. It's wise to switch to alternatives when:
- Generation speed drops below 1 to 2 tokens per second. This is slower than the average reading speed, making working with the model tiring.
- The system becomes unstable due to overheating or shuts down spontaneously.
- The required tasks demand a larger model than your memory can physically hold.
In those cases, it may be wiser to consider privacy-friendly API services or to run the AI tasks on another, more powerful device on your local network.
Optimization checklist for older hardware
Use these steps as a quick reference card when setting up your local model:
- Check available RAM: Open Task Manager or Activity Monitor and determine how much memory is actually free.
- Choose a suitable model: Select a 1.5B or 3B model (for example, Qwen2.5-1.5B or Llama-3.2-3B).
- Select strong quantization: Download a Q4_K_M or Q3_K_M variant of the model.
- Limit the context: Manually set the context length to 1024 or 2048 tokens in the settings.
- Close other software: Close all browser windows and heavy background programs.
- Measure performance yourself: Test the model with a short prompt and check the generation speed and temperature trend of your machine.
