Compiling llama.cpp with CUDA for maximum performance

Along the route of running language models locally (choosing, installing, using, connecting, and managing), this article sits in the installation and performance optimization phase. Here we look at the step from ready-made distributions to a manually compiled runtime. If you want to determine in advance which components are suitable, you can check what hardware local LLMs require to see whether a specific graphics card offers enough memory bandwidth. This article covers building llama.cpp from source with Nvidia CUDA acceleration.

Widely used applications rely on llama.cpp under the hood, but often ship generically compiled binaries to guarantee broad compatibility across multiple GPU generations. By compiling the source code yourself on a Linux system, we tune the generated machine code to the exact compute architecture of the graphics card present. That avoids generic runtime fallbacks, exploits specific hardware instructions, and delivers more stable token throughput.

Why manual compilation is faster than prebuilt binaries

Generic binary distributions of inference engines are built with a broad audience in mind. To avoid errors on older graphics architectures, specific microcode optimizations are often disabled or handled through dynamic intermediate layers. When we supply the flag -DCMAKE_CUDA_ARCHITECTURES during configuration, the NVCC compiler compiles targeted binary instructions directly for the streaming multiprocessors of that particular graphics card.

Compiling yourself also makes it possible to build specific hardware accelerators straight into the build. Think of specialized Flash Attention kernels that reduce prompt processing latency with larger context windows, or cuBLAS bindings that distribute matrix operations efficiently. With large input documents, a hardware-specific build produces a noticeably shorter processing time before the first token appears.

An added advantage is avoiding superfluous abstraction layers. Anyone setting up a dedicated server environment often has no need for background services or graphical management interfaces. With standalone binaries such as llama-cli and llama-server you keep full control over memory management, threads, and layer distribution without the involvement of external daemons.

Preparing requirements and the CUDA Toolkit on Linux

For a stable build environment we assume a common Linux installation. More on the basic configuration of the operating system and drivers can be found in the overview of running local LLMs on Linux. Always check that the official Nvidia drivers are loaded before you begin installing development packages.

Check from the terminal whether the driver is active and which driver version is loaded:

nvidia-smi

Next we install the required development packages through the distribution's package manager: git, cmake, a C++ compiler (such as build-essential on Debian-based systems) and the Nvidia CUDA Toolkit. For distribution-specific package names and repositories, always consult the documentation of your distribution and Nvidia:

sudo apt update && sudo apt install -y \
  git \
  build-essential \
  cmake \
  libcurl4-openssl-dev \
  nvidia-cuda-toolkit

After installation, check that the CUDA compiler (NVCC) can be invoked correctly:

nvcc --version

Should the terminal report that nvcc cannot be found, the associated binary paths have to be added to the environment path manually (through ~/.bashrc):

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Determining the right Nvidia compute capability

An essential parameter during configuration with CMake is the compute capability (CC) of the GPU. This numeric value defines which generation of hardware instructions the compiler may generate. Supplying the exact architecture prevents generic PTX intermediate code that would otherwise have to be compiled when the model starts.

With an Ampere card, then, you choose architecture 86. If a system has graphics cards from several generations (a Turing and an Ampere card in the same machine, for example), multiple architectures can be specified separated by a semicolon, such as 75;86.

Fetching the source and building with CMake, step by step

We fetch the llama.cpp source from the official GitHub repository. Then create a separate build directory to keep the compiled files apart from the source files:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build

Next run CMake with the CUDA parameters enabled. The flag -DGGML_CUDA=ON activates the Nvidia backend. In the example below we set the architecture to Ampere (CC 86) and enable Flash Attention for quantized models:

cmake .. \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DCMAKE_BUILD_TYPE=Release

With -DGGML_CUDA_FA_ALL_QUANTS=ON Flash Attention is compiled for common quantization formats, which helps limit memory use with longer contexts. Then start the compilation across all available processor cores:

cmake --build . --config Release -j$(nproc)

When the build process finishes, the binaries are in the directory bin/. The most commonly used executables are llama-cli for interactive terminal prompts, llama-server for serving an HTTP endpoint, and llama-bench for measuring compute performance.

Performance measurement and verification with llama-bench

To establish that the compiled software actually uses the CUDA cores, we start the built-in benchmark tool with a GGUF model file. A detailed explanation of how compression formats such as Q4_K_M affect memory footprint and precision can be found in the explanation of quantization explained for local models.

Run the benchmark program with a test model to evaluate prompt processing and generation speed:

./bin/llama-bench \
  -m ../models/model.gguf \
  -n 128 \
  -p 512 \
  -ngl 99

With the parameter -ngl 99 (number of GPU layers) we indicate that all model layers should be sent to video memory. The terminal report immediately shows which CUDA device has been selected for the computations.

For a methodical analysis of the results and a clear distinction between processing time and generation time, you can consult the concepts in the article on measuring latency, throughput, and tokens per second. During analysis, watch two primary measurements:

• Prompt processing (pp): This indicates how many tokens per second the engine processes while reading in the input prompt. This phase depends mainly on the compute power of the GPU (compute-bound).
• Text generation (tg): This measures the number of tokens generated per second while producing the answer. This phase is primarily limited by the memory bandwidth of the graphics card (memory-bandwidth bound).

Critical runtime parameters for the server and CLI

Final token throughput depends not only on compilation but also on the settings used to start the server. For continuous deployment we usually start the HTTP server interface with appropriate parameters:

./bin/llama-server \
  -m ../models/model.gguf \
  -c 8192 \
  -ngl 99 \
  --flash-attn \
  --threads 8 \
  --host 127.0.0.1 \
  --port 8080

The most important parameters explained:

• -ngl / --n-gpu-layers: The number of transformer layers sent to the GPU. If a model does not fit entirely in VRAM, some of the layers can be moved while the rest runs on the processor.
• -c / --ctx-size: The maximum context size in tokens. A larger context requires more memory space for the key-value (KV) cache.
• --flash-attn: Activates Flash Attention, which limits memory growth at larger context sizes and speeds up prompt evaluation.
• --threads: The number of CPU threads used for operations outside the GPU. It is advisable to set this equal to the number of physical compute cores of the processor.

Memory distribution and bottlenecks in hybrid offloading

When a model is larger than the available video memory, llama.cpp offers the option of placing some layers on the GPU and assigning the remaining layers to the CPU and system memory. Although this allows heavier models to be loaded on modest hardware, it introduces a considerable slowdown.

While tokens are being generated, intermediate results have to be exchanged over the PCIe bus between video memory and system RAM at every step. Total throughput is therefore limited by the slowest link: the bandwidth of system memory and the PCIe interface. As a result, the final token speed in a hybrid configuration is markedly lower than when a model fits entirely within video memory.

In practice, a slightly more compact quantized model that fits entirely in VRAM often delivers a far more responsive interaction than a larger model that has to be split across CPU and GPU.

Troubleshooting compilation and runtime

Specific error messages can occur when building and starting compiled software with GPU support. Below are solutions for common situations.

1. CUDA out of memory (OOM)

When startup halts with a memory error, the combination of model parameters, context size, and KV cache exceeds physical VRAM capacity. This can be resolved by lowering the number of offloaded layers (-ngl) or choosing a smaller context length (-c). Check with nvidia-smi whether other applications are unnecessarily occupying video memory.

2. CMake does not detect the CUDA compiler

If CMake reports that CMAKE_CUDA_COMPILER cannot be found, the development headers are missing or the path to NVCC is not in the environment variables. In that case, pass the full path to the CMake configuration:

cmake .. -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc

3. Compiler incompatibilities

When using very recent GCC versions on a distribution, the CUDA Toolkit may report an error about an unsupported GNU compiler version. In that case, install a specifically supported compiler version (such as GCC 12) and point CMake to it with parameters:

cmake .. -DGGML_CUDA=ON \
  -DCMAKE_C_COMPILER=gcc-12 \
  -DCMAKE_CXX_COMPILER=g++-12 \
  -DCMAKE_CUDA_ARCHITECTURES=86

Privacy and network security when running locally

An important reason to build software from source yourself and host it locally is full control over data and privacy. When running a local llama-server , prompts, documents, and model answers never leave your own system. No diagnostic data or telemetry requests are sent to external cloud services. How to set this up within organizational frameworks is described in the article on privacy-friendly AI use.

To make sure the server interface is not unintentionally exposed to others on the local network, it is wise to have the service listen solely on the loopback address (--host 127.0.0.1), unless a secured proxy or VPN layer for authentication has been placed in front of it.

Maintenance and incremental updates

The llama.cpp source is developed continuously with new compute kernels and support for recent model architectures. There is no need to repeat the full configuration process for updates. Fetching the latest source and repeating the build command is enough:

cd llama.cpp
git pull
cd build
cmake --build . --config Release -j$(nproc)

Thanks to its incremental structure, CMake compiles only the files that have actually changed. That keeps the build environment quick to update and makes new optimizations immediately available for local language models.