When it does not work: out-of-memory, slow tokens and a GPU that stays out of it

By Ivo Donker — compiled with AI assistance · August 7, 2026

Every installation guide ends at "it works", but that is where the real work starts: models that will not load, output that is agonizingly slow, or a graphics card that stays idle. This guide is the troubleshooting article for local LLM setups: symptom, diagnosis, solution; checked on 2026-08-07. Within the canon of the guide, this page sits in pillar 2 ("Hardware, performance & energy"). Within the general route of the guide — choosing, installing, using, connecting, and managing or serving — this page follows directly on the hardware phase. If your machine falls structurally short, the page hardware for local LLMs helps you determine the basic requirements, while the guide optimizing LLMs on older hardware gives you options for older systems.

Commands, configuration files and settings in tools such as Ollama, LM Studio and llama.cpp can change from one software version to the next. It is therefore important to consult the documentation of your specific client if the graphical shell or the command line interface differs from the examples in this article. Also note that the check date of 2026-08-07 serves as the reference point for the current generation of open-weights models and runtime engines.

Hardware floor as a starting point

The examples in this article assume a number of realistic hardware configurations that serve as a reference for your own installation. A common starting point is a system with 16 GB of working memory combined with a model of 7 to 8 billion parameters (7-8B) in a 4-bit quantization (Q4), which comes down to about 5 GB of memory load. This runs on Windows or Linux with a dedicated graphics card with enough VRAM, or on an Apple Silicon Mac with unified memory. For anyone who wants to experiment quickly or has more limited resources, 8 GB of working memory with a more compact model of 3 to 4 billion parameters is the floor.

When you notice that your hardware sits far below these specifications and the model simply refuses to work, it is wise to adjust your expectations or switch to an alternative configuration. You will find a thorough analysis of what your machine needs at a minimum on the page about hardware for local LLMs. The goal of this guide is not to make impossible hardware dreams come true, but to solve the bottlenecks in your current local workflow systematically.

Step 0: diagnosis — measure before you change

Before you adjust settings at random, rewrite configuration files or delete models, you have to establish exactly what is happening under the hood. Many users get frustrated because they fight symptoms instead of the actual cause. A systematic diagnosis starts with answering three core questions: does the model load into memory at all, how much memory (RAM and VRAM) is in use at that moment, and which hardware component (CPU, GPU or memory bus) is doing the calculations?

To get an objective picture of performance, you can use a structured measurement protocol. That keeps you from relying on subjective impressions such as "it feels slow". Compare your setup in a fair way using the pointers on comparing models on your own hardware. It is also useful to look at the distribution of the latency instead of leaning on a single average; for that you can turn to the measurement protocol for measuring latency percentiles on the networked benchmark subdomain.

During the diagnosis you check the active processes via Task Manager on Windows, Activity Monitor on macOS or commands such as top and nvidia-smi on Linux. Look specifically at the temperature of your processor and graphics card. If the temperatures rise to the maximum, thermal throttling kicks in and the hardware falls back in clock speed to protect itself, which results in extreme slowness.

Symptom 1: out-of-memory (OOM)

Out-of-memory situations are among the most common frustrations when running language models locally. It comes down to the requested amount of data being larger than the physical capacity of your RAM or VRAM, after which the operating system or the runtime engine intervenes abruptly.

Recognizing OOM

You recognize this symptom by hard error messages in your console or log files. Common messages are "CUDA out of memory", "cannot allocate memory", or a sudden segmentation fault on Linux. Sometimes the application crashes right when the model file is loaded, or you see the system become extremely slow because the working memory fills up and the computer starts writing data en masse to the slow hard disk or SSD (paging or swapping).

Diagnosing OOM

Open your task manager or activity monitor while the model is starting up. Do you see the memory percentage shoot to 99 percent in a fraction of a second, followed by a crash? Or do you run a command such as ollama ps to see which processes are active and how much memory they claim? Often you see that the model does try to load, but strands halfway through the layers because there are simply no free gigabytes left.

Solutions for OOM in order of impact

1. Choose a smaller model or a smaller quantization: If your current model file is too large for your hardware, switch to a variant with fewer parameters or heavier compression. You can read how quantization works and which quality trade-offs come with it on quantization explained. If you really do not know which size fits, consult choosing a local model.
2. Reduce the context length: A larger memory window eats gigabytes of working memory during inference. By making the window smaller, you save memory immediately. This is covered extensively on optimizing the context window locally.
3. Move layers to the CPU: If your graphics card has too little VRAM, you can set part of the model layers to be handled by the regular processor and the working memory. This does slow down the output, but it prevents a total crash.
4. Accept the hardware limit: Sometimes a model simply does not fit. If no setting helps, the model is too large for your machine. In that case you can turn to hardware for local LLMs to see what you really need.

Symptom 2: slow tokens

Slow tokens mean that the model does answer, but that the speed per second lags far behind what is acceptable for interactive use. Where a smooth local setup quickly generates dozens of tokens per second, a misconfigured system can drop to less than one word per second.

Recognizing slow tokens

You notice this because words appear on the screen agonizingly slowly, as if someone is typing them one by one. This is subjective as long as you do not measure, but as soon as you notice that a simple sentence needs more than a minute to finish, there is a performance problem. To record this objectively and to check whether your optimizations have any effect, you can use the guidelines on measuring speed.

Causes of slow tokens

The most common cause is CPU fallback: the software has forgotten to switch on the graphics card, which means the slow CPU has to do all the calculations on its own. Other causes are a gigantic context length that overloads the working memory, thermal throttling with a howling fan, or hitting the memory bandwidth limit on integrated graphics chips, as described in hardware for local LLMs and optimizing LLMs on older hardware.

Solutions for slow tokens

1. Measure before and after: Run a fixed benchmark so that you know whether a change has any effect. For this, consult measuring speed.
2. Reduce the context: Long documents in the prompt cause heavy calculations per generated token. Optimize this via optimizing the context window locally.
3. Enable speculative decoding: If your runtime engine supports this, you can use a small helper model to propose tokens that the large model then approves in one go.
4. Set realistic expectations: Older hardware has physical limits. If you work with legacy equipment, look at the optimization strategies on optimizing LLMs on older hardware.

When you are dealing with API integrations in which slow local responses occur, you also have to take measures on the application side. For this you can consult the guidelines on timeouts and cancellation to make sure your software does not hang on slow answers.

Symptom 3: the GPU that stays out of it

A persistent problem is that you do have a powerful graphics card in your computer, but the LLM software makes no use of it at all. The model runs entirely on the processor, which makes performance disappointing.

Recognizing a sleeping GPU

You see this because the generation speed is very low, while your task manager indicates that the CPU is running at 100 percent and the graphics card is watching idly. Commands such as ollama ps show that the model has been loaded entirely on the CPU, and utilities such as nvidia-smi on Windows or Linux show no active LLM processes in the VRAM.

Causes of missing GPU acceleration

Causes range from missing or outdated drivers and a Metal interface that is not switched on in macOS to a miscommunication about CUDA versions on Windows or Linux. It also happens that the user has not explicitly told the settings of the client (such as LM Studio or Ollama) how many layers should be sent to the graphics card.

Platform-specific check overview

Use the table below to check per operating system where you have to look and which action you can take:

Common mistakes when troubleshooting

When solving problems, users often fall into the same traps. Avoiding these mistakes saves you hours of frustration:

• Changing too many variables at once: Never adjust the quantization, the model, the context length and the hardware settings at the same time. If it then suddenly works, you do not know which change solved the problem. Change one parameter at a time and test again.
• Forgetting that the theoretical background is settled: Do not try to reinvent why context costs memory or how quantization works; instead, consult the factual documentation such as KV caching architecture.
• Staring at averages instead of outliers: An average speed says little if your model stalls completely now and then. Always look at the distribution of the performance.
• Ignoring power consumption and heat: Local AI draws a lot of energy. A server running continuously quickly swallows dozens of watts, which leads to heat problems. So check your energy budget via power consumption for local AI.
• Fixating on English benchmarks with Dutch input: Models respond differently to Dutch texts. Always test with realistic local input. An example of this is a slow Dutch summary that suddenly comes in at an acceptable speed after the context has been reduced or the GPU has been switched on. If you want to optimize the quality of the Dutch output, take a look at better Dutch.

When you should stop troubleshooting

There comes a moment when further troubleshooting no longer makes sense. If your hardware is structurally too small for the tasks you want to run — something you can verify via hardware for local LLMs or optimizing LLMs on older hardware — then every attempt remains mopping the floor with the tap running. A model that does not fit because of physical limits will never run stably.

In such situations it is wise to make up your mind and choose an alternative route. You can switch to an externally offered API service. To make sure your application keeps working robustly when your local setup unexpectedly fails or falls short, you can build in fallbacks using the manual on robust integrations.

Privacy and energy when running locally

One of the main motives for running LLMs locally is safeguarding your privacy. With a correctly configured local setup, no prompt, document or generated answer leaves your own machine; all calculations take place locally within your own network domain. You can read more about this on privacy-friendly AI.

In addition, energy plays a crucial role in long-term use. Continuously loading a graphics card or processor causes considerable power consumption that can run up to a few dozen or even hundreds of watts per hour, depending on your hardware. Anyone who wants to keep a grip on the energy costs of local servers running around the clock does well to go through the insights on power consumption for local AI .

Checked on 2026-08-07.