# Installing Kobold.cpp for local GGUF inference

[Skip to content](#lm-inhoud)Network/[NL](/en/kobold-cpp-installeren-voor-lokale-gguf-inferentie)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkobold-cpp-installeren-voor-lokale-gguf-inferentie&text=Installing%20Kobold.cpp%20for%20local%20GGUF%20inference)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkobold-cpp-installeren-voor-lokale-gguf-inferentie)[](https://www.reddit.com/submit?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkobold-cpp-installeren-voor-lokale-gguf-inferentie&title=Installing%20Kobold.cpp%20for%20local%20GGUF%20inference)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkobold-cpp-installeren-voor-lokale-gguf-inferentie&text=Installing%20Kobold.cpp%20for%20local%20GGUF%20inference)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkobold-cpp-installeren-voor-lokale-gguf-inferentie)[](https://www.reddit.com/submit?url=https%3A%2F%2Fgids.llmnet.nl%2Fen%2Fkobold-cpp-installeren-voor-lokale-gguf-inferentie&title=Installing%20Kobold.cpp%20for%20local%20GGUF%20inference)[](#)

 
# Installing Kobold.cpp for local GGUF inference

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 
 Hardware floor for this guide: Recommended configuration for this guide: a system with 16 GB of system RAM, 8 GB of VRAM (an Nvidia RTX 3070 or RTX 4060, for instance) and an 8-core CPU, running Windows 11 or Ubuntu 24.04 LTS. As an example model we use Llama-3.1-8B-Instruct.Q4_K_M.gguf (roughly 4.92 GB file size, requiring around 6.5 GB of allocated VRAM for an 8k context). Version: Kobold.cpp v1.118.1 (released on 2 August 2026).

 

 Along the path of choosing → installing → using → connecting → managing this guide sits squarely at the second step: installing per platform. Before you start with inference software, it is essential to check that your workstation meets the minimum memory specifications. Consult the overview of [what hardware you need to run LLMs locally](https://gids.llmnet.nl/en/hardware-voor-lokale-llm) beforehand to verify that your combination of system memory and graphics card is sufficient for the model size you have chosen.

 Kobold.cpp is a notable exception within the local language model ecosystem. Where many solutions depend on heavy Python environments, Docker containers or background daemons, Kobold.cpp delivers a compact C/C++ implementation that runs as a single executable. The project combines the compute power of llama.cpp with a built-in web interface, an extensive OpenAI-compatible API, and advanced memory management features such as context shifting and smart context. That makes it a particularly stable foundation for anyone who wants to work with GGUF files directly, without overhead.

 
## What Kobold.cpp is and how it differs from other engines

 Kobold.cpp is a standalone inference engine designed primarily to run GGUF models locally, quickly and with minimal system requirements. Originally created as a backend for creative writing and interactive fiction, the package has grown into a full-fledged local server for general chat, document analysis and API-driven automation.

 The structural difference with engines such as Ollama or vLLM lies in the architecture. Ollama abstracts many settings away behind a daemon and manages its own layered model manifests, while vLLM focuses on batch processing for dozens of concurrent users on Linux servers. Kobold.cpp, by contrast, targets a single user or a small workgroup, with the user retaining full control over the allocation of VRAM, threads and context management. Kobold.cpp also ships a zero-dependency executable: on Windows you download a .exe that starts straight away, with no installation process and no administrator rights.

 The file format at the center of it all is GGUF. To get to grips with the details of this format and understand how weights are compressed with minimal loss of precision, read the basics in the guide to [quantization and running large models on small hardware](https://gids.llmnet.nl/en/kwantisatie-uitgelegd). The network file on [choosing between GGUF, AWQ and EXL2 quantization formats](https://hub.llmnet.nl/en/quantization-formaten-kiezen-gguf-awq-of-exl2-vergeleken) also explains precisely why GGUF is the best choice when you want to distribute model weights flexibly across both the GPU and regular system memory.

 
 
 
 
 Property | 
 Kobold.cpp | 
 Ollama | 
 vLLM | 
 

 
 
 
 Installation type | 
 Standalone executable (portable) | 
 System daemon / CLI | 
 Python / CUDA wheel | 
 

 
 Default interface | 
 Built-in web UI (Kobold Lite) | 
 None (a separate Open WebUI is required) | 
 None (pure API server) | 
 

 
 CPU/GPU offloading | 
 Adjustable layer by layer (GPU offload) | 
 Automatic / Modelfile | 
 Primarily full GPU allocation | 
 

 
 Memory management | 
 Context shift & smart context | 
 Recomputation on overflow | 
 PagedAttention (virtual memory) | 
 

 
 File format | 
 GGUF (opened directly from disk) | 
 Proprietary blobs (GGUF wrapper) | 
 Hugging Face safetensors / AWQ / GPTQ | 
 

 
 
 

 
## Privacy and data protection when running locally

 One of the main reasons to run inference locally through Kobold.cpp is absolute control over incoming and outgoing data flows. When Kobold.cpp runs on your computer, all computation takes place on your own processor and graphics card. There is no telemetry, no prompts are sent to external cloud services, and no internet connection is needed for the model to work.

 For organizations that have to comply with strict privacy legislation, this setup offers full transparency. The document on [privacy-friendly AI use](https://gids.llmnet.nl/en/privacyvriendelijk-ai) goes deeper into the legal and infrastructural safeguards needed when you process personal data or trade secrets locally. By default, Kobold.cpp binds its web server exclusively to 127.0.0.1 (localhost), so the port is not open to third parties on the same local network without explicit flags.

 
## Installation and downloads per platform

 Because Kobold.cpp is written in optimized C/C++, precompiled binaries are available for Windows, Linux and macOS. Below we cover the installation steps for each operating system.

 
### Windows (Nvidia CUDA, AMD ROCm or CPU)

 On Windows the installation is simplest:

 1. Go to the official GitHub releases of koboldcpp and download the executable koboldcpp.exe (for general use and Nvidia GPUs) or koboldcpp_rocm.exe (for specific AMD Radeon cards).

 2. Put the file in a fixed folder, for example C:\AI\KoboldCPP\.

 3. Double-click the .exefile. A graphical configuration screen (the launcher) appears, in which you can select your settings and the GGUF file to load.

 
### Linux (manual or compiled with CUDA)

 On Linux you can choose between downloading a prebuilt binary and compiling it yourself for maximum hardware optimization. Building it yourself with CUDA support often yields better performance:

# Installeer de basisafhankelijkheden
sudo apt-get update && sudo apt-get install -y build-essential libopenblas-dev

# Clone de repository
git clone https://github.com/LostRuins/koboldcpp.git
cd koboldcpp

# Compileer met Nvidia CUDA-versnelling
make LLAMA_CUDA=1 -j$(nproc)

# Start Kobold.cpp via de command-line
./koboldcpp.py --model /pad/naar/model.gguf --gpulayers 33 --contextsize 8192

 
### macOS (Apple Silicon Metal)

 On a Mac with Apple Silicon (M1/M2/M3/M4), Kobold.cpp uses Metal for acceleration through the integrated unified memory:

# Clone en compileer met Metal-ondersteuning
git clone https://github.com/LostRuins/koboldcpp.git
cd koboldcpp
make LLAMA_METAL=1 -j8

# Start met Metal offloading
./koboldcpp.py --model /pad/naar/model.gguf --usemetal --contextsize 8192

 
## The graphical interface versus command-line flags

 When you start the launcher, you can configure the engine by hand through the graphical controls or start it directly with parameters. For repetitive tasks or headless servers, it is convenient to create a startup script.

 Below is an overview of the most important configuration parameters and what they do:

 
 
 
 
 CLI parameter | 
 GUI name | 
 Function and recommended value | 
 

 
 
 
 --model | 
 Model file | 
 Path to the .gguffile on your local disk. | 
 

 
 --gpulayers / --ngl | 
 GPU layers | 
 The number of layers moved into VRAM (33 for an 8B model, for instance). | 
 

 
 --threads | 
 Threads | 
 The number of physical CPU cores computing the CPU layers (usually 4 to 8). | 
 

 
 --contextsize | 
 Context size | 
 The length of the context window in tokens (4096 or 8192 by default). | 
 

 
 --usecublas / --usevulkan | 
 Hardware backend | 
 Determines the acceleration: cuBLAS (Nvidia), Vulkan (universal/AMD) or Metal (Apple). | 
 

 
 --port | 
 Port | 
 The network port for the web UI and API (default 5001). | 
 

 
 --smartcontext | 
 Smart context | 
 Prevents needless recomputation of prompt tokens in long conversations. | 
 

 
 
 

 
## Memory allocation and GPU offloading: how layers work

 A transformer model consists of a series of successive neural network layers. An 8B model such as Llama-3.1 has 32 decoder layers plus an input and an output layer (33 layers in total). Kobold.cpp lets you decide exactly how many of these layers run on the graphics card (VRAM) and how many run in system memory (RAM, through the CPU).

 When you offload all 33 layers to the GPU using --gpulayers 33, generation runs at maximum speed. If your graphics card has only 6 GB of VRAM and the model needs 8 GB, however, you can set up a hybrid split: 20 layers on the GPU and 13 on the CPU, for example. Token speed then drops considerably compared with full GPU processing, but the model starts and runs without stalling.

 To calculate how much VRAM you need, take two components into account: the base model and the KV cache (the context memory). An 8B Q4_K_M model takes up roughly 4.9 GB for the weights. For a context length of 8192 tokens, an FP16 KV cache adds about another 1.5 GB. For a deeper look at tuning your context length correctly to avoid running out of memory, read the background article on [setting the context length when running locally](https://gids.llmnet.nl/en/context-window-optimaliseren-lokaal).

 
## A worked example: loading and testing a GGUF model

 Let us walk through a concrete startup command for a system with an Nvidia RTX 4060 (8 GB VRAM) and 32 GB of RAM, loading an 8B instruction model:

# Start Kobold.cpp met cuBLAS versnelling en 8k context
koboldcpp.exe --model C:\LLM\Llama-3.1-8B-Instruct.Q4_K_M.gguf \
 --usecublas normal \
 --gpulayers 33 \
 --contextsize 8192 \
 --threads 6 \
 --smartcontext \
 --port 5001

 As soon as the terminal reports that the model has loaded into memory successfully, the built-in web interface opens automatically in your browser at http://127.0.0.1:5001. Kobold Lite offers immediate access to two modes: story mode (free text generation) and chat mode (question-and-answer interaction with adjustable instruction systems).

 
## Context management: context shifting and smart context

 One unique technical advantage of Kobold.cpp over many other tools is the mechanism called context shifting. With traditional inference servers, the entire KV cache is wiped and recomputed from scratch as soon as a conversation grows longer than the configured context window. In long sessions this causes a sudden freeze of several seconds or even minutes.

 On reaching the context limit, context shifting removes only the oldest tokens from the middle of the KV cache and shifts the remaining tokens forward. The initial system prompt is preserved, the most recent messages stay intact, and the GPU only has to perform a minimal correction. Responses therefore keep starting immediately, however long the chat session runs.

 
## Connecting external applications through the API

 Kobold.cpp functions not only as a standalone chat environment but also as a backend server for external interfaces such as SillyTavern, Jan AI, AnythingLLM or your own Python scripts. The server exposes two API endpoints:

 1. The Kobold native endpoint (http://127.0.0.1:5001/api/v1/generate), which gives specialized interfaces access to in-depth sampler settings such as Mirostat, tail-free sampling and repetition penalties.

 2. The OpenAI-compatible endpoint (http://127.0.0.1:5001/v1/chat/completions), which lets you use Kobold.cpp as a drop-in replacement in software designed by default for OpenAI models.

 Below is a minimal Python example that talks to Kobold.cpp's local OpenAI endpoint:

import urllib.request
import json

url = "http://127.0.0.1:5001/v1/chat/completions"
headers = {"Content-Type": "application/json"}
payload = {
 "model": "koboldcpp",
 "messages": [
 {"role": "system", "content": "Je bent een deskundige en feitelijke assistent."},
 {"role": "user", "content": "Leg in twee zinnen uit wat het voordeel is van een lokale LLM."}
 ],
 "temperature": 0.7,
 "max_tokens": 150
}

req = urllib.request.Request(url, data=json.dumps(payload).encode('utf-8'), headers=headers)
with urllib.request.urlopen(req) as response:
 result = json.loads(response.read().decode('utf-8'))
 print(result["choices"][0]["message"]["content"])

 
## Testing performance on Dutch instructions

 To verify that inference is running correctly and the samplers are set up properly, it is advisable to test a structured Dutch-language prompt. Here we assess both logical consistency and token speed (tokens per second).

 Test input:

 "Give a point-by-point comparison of synchronous and asynchronous network requests in web applications. Focus on throughput, complexity and memory usage."

 Indicative order of magnitude (Llama-3.1-8B-Instruct, Q4_K_M on an RTX 4060):

 As an indicative figure, the model generates a structured response at a processing speed of roughly 40 to 55 tokens per second. Prompt evaluation (reading in the input) runs at several hundred tokens per second thanks to cuBLAS flash attention, although actual speed depends heavily on the specific graphics card, driver version, quantization and context length. The output shows correct Dutch grammar and a clear separation between the requested criteria.

 
## Troubleshooting and known bottlenecks

 Despite the stability of Kobold.cpp, situations can arise in which the model does not start or performs unexpectedly slowly. If you run into persistent error messages, see the extensive troubleshooter for [out-of-memory messages, slow tokens and GPU detection problems](https://gids.llmnet.nl/en/als-het-niet-werkt-out-of-memory-trage-tokens-en-een-gpu-die-niet-meed).

 The most common operational bottlenecks and their immediate fixes:

 1. CUDA out of memory (OOM): Layer allocation exceeds physical VRAM. Lower --gpulayers by 2 to 4 steps or reduce the --contextsize from 8192 to 4096 tokens, for instance.

 2. Extremely low generation speed (1-2 tokens/s): This indicates that the model is running entirely on the CPU, or that memory is swapping between RAM and disk. Check that the right backend (cuBLAS, ROCm or Metal) is active and that --usecublas has been passed.

 3. Crashes when loading large contexts: Enable the flag --no-mmap if your disk responds slowly or if the operating system's virtual memory fills up while mapping the GGUF file.

 
## Conclusion and next steps in your local setup

 Kobold.cpp offers one of the most reliable and lightweight ways to run GGUF language models locally. The absence of complex background services and the direct control over GPU offloading and context shifting make it an excellent foundation for beginners and advanced users alike who want to keep maximum control over their hardware.