Running Local LLMs on Linux

If you want to get serious about running Large Language Models (LLMs) locally, Linux is the absolute best operating system. While Windows via WSL2 or macOS via Metal are certainly capable, Linux offers direct and unfiltered access to your hardware. This means you get maximum performance from your graphics card (GPU), lose less system memory to background processes, and can easily convert an old gaming PC in a closet into a powerful AI server.

In this comprehensive guide, we will cover everything you need to know to get started locally. We will discuss installing the correct drivers for both NVIDIA and AMD, and look at two of the most popular methods for running models: the user-friendly Ollama and the flexible llama.cpp.

1. System Preparation and Hardware Check

Before installing software, it is essential to know what hardware we are working with. For local AI models, the amount of VRAM (Video RAM) on your graphics card is the main limiting factor. Read more about the ideal hardware requirements in our article on hardware for local LLMs.

Start by updating your system, preferably on a clean installation of Ubuntu 22.04 LTS or Debian 12:

sudo apt update && sudo apt upgrade -y

Next, check which graphics card is recognized by the system using the following command:

lspci | grep -i vga

This will tell you immediately whether your system registers an NVIDIA, AMD, or integrated Intel chip. Make a note of this, as it determines your next step when installing the drivers.

2. Installing Graphics Drivers (GPU Acceleration)

Running a model purely on the processor (CPU) is possible, but painfully slow. To achieve acceptable generation speeds (tokens per second), the model must be partially or fully offloaded to the GPU. For this, you need the correct computational drivers: CUDA for NVIDIA, or ROCm for AMD.

NVIDIA (CUDA Toolkit)

NVIDIA is currently the industry standard for AI. Fortunately, installing the correct drivers on Ubuntu is now straightforward using the built-in driver manager.

1. Install the proprietary NVIDIA drivers (choose version 535 or newer):

   sudo ubuntu-drivers autoinstall

2. Restart your machine: sudo reboot
3. Next, install the NVIDIA CUDA toolkit so that software like llama.cpp can communicate directly with the Tensor cores:

   sudo apt install nvidia-cuda-toolkit

4. Verify the installation by checking the status:

   nvidia-smi

When you run nvidia-smi, you will see a table. The most important value for LLMs is in the "Memory-Usage" column (for example: 0MiB / 24576MiB). The more VRAM you have free, the larger the model you can load.

AMD (ROCm)

AMD cards (especially the RX 6000 and 7000 series) are excellent, more affordable alternatives, but the software stack is more complex. For this, we use ROCm (Radeon Open Compute).

Download the ROCm installation script from the official AMD website for your specific Ubuntu version. Make sure to install the amdgpu-install tool with specific use cases:

amdgpu-install --usecase=rocm,hip,mivisionx

A crucial part of AMD on Linux is the permission structure. Add your current user to the video and render groups, otherwise the software will not be able to access the GPU:

sudo usermod -a -G video $USER
sudo usermod -a -G render $USER

3. Method 1: Installing Ollama (The Easy Route)

If you want quick results and do not want to compile anything yourself, Ollama is the best choice. Ollama bundles llama.cpp and all necessary dependencies into a simple system that feels like Docker, but specifically for local AI.

Installation via the Script

You can install Ollama with a single simple curl command in your terminal:

curl -fsSL https://ollama.com/install.sh | sh

This script automatically detects whether you have an NVIDIA or AMD GPU, downloads the correct binaries, and sets up Ollama as a background service via `systemd`. This means Ollama will always run in the background as soon as your server boots up.

Running Your First Model

Once the installation is complete, you can immediately download and start a model. Meta's Llama 3 is currently an excellent choice for general use:

ollama run llama3

The first time you run it, Ollama will download the model file (several gigabytes, depending on the model). Once this is done, you will get a prompt in your terminal where you can ask questions directly.

Configuring Ollama for Network Access (Headless Server)

By default, Ollama only listens to connections on the local machine (localhost/127.0.0.1 on port 11434). If your Linux machine is sitting in a closet somewhere without a monitor (headless), and you want to access the AI from your laptop, you need to adjust the systemd configuration.

1. Open the systemd editor for Ollama:

   sudo systemctl edit ollama.service

2. Add the following lines in the editor, which specifies that Ollama should listen on all network interfaces:

   [Service]
   Environment="OLLAMA_HOST=0.0.0.0"

3. Reload the systemd daemon and restart Ollama:

   sudo systemctl daemon-reload
   sudo systemctl restart ollama

Now you can access Ollama via the IP address of your server on your home network (e.g., http://192.168.1.50:11434).

4. Method 2: Installing Llama.cpp (For Maximum Control)

While Ollama is fantastic for beginners, it does not always offer the latest features or full control over inference parameters. For purists, running llama.cpp directly is recommended. This package is written in pure C/C++ and is extremely efficient.

Compiling from Source

Since every Linux installation has slightly different hardware, it is recommended to compile llama.cpp yourself on your machine. For this, we need build-essential and git.

sudo apt install build-essential git cmake
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

Now we need to compile the project. Please note: The default make command builds a CPU-only version. To enable GPU acceleration, you must pass specific flags.

For NVIDIA (CUDA):

make LLAMA_CUDA=1

For AMD (ROCm/HIP):

make LLAMA_HIPBLAS=1

Compiling may take a few minutes. Once completed, you will find a file named llama-server in the folder.

Downloading and Running a GGUF Model

Llama.cpp uses the GGUF file format. This is a special file format in which the model is "quantized" (compressed) so that it fits into the RAM and VRAM of consumer hardware. For more depth on this, we recommend reading our article quantization explained.

You can download models via platforms like Hugging Face. Once you have a .gguf file (for example, mistral-7b-v0.1.Q4_K_M.gguf), you can start the server:

./llama-server -m mistral-7b-v0.1.Q4_K_M.gguf -c 4096 -ngl 99 --host 0.0.0.0

• -m: The path to your GGUF model.
• -c 4096: Defines the context size (how much text the model can 'remember' in a single conversation).
• -ngl 99: This is crucial. This stands for "Number of GPU Layers". By setting this to 99, you force llama.cpp to offload all layers of the model to the graphics card (assuming you have enough VRAM).
• --host 0.0.0.0: Opens the server to your local network.

5. Usage and Integration into Your Workflow

A running model in your terminal or as a background service is impressive, but not very user-friendly. Fortunately, there are countless ways to interact with your local LLM.

Web Interfaces (Graphical UI)

The most popular method is setting up a web interface similar to ChatGPT. The absolute top recommendation for this is Open WebUI. This interface runs as a Docker container and seamlessly connects to your Ollama installation or llama.cpp server. You can upload documents, save conversation history, and compare different models side-by-side. Read our step-by-step guide on setting up Open WebUI for full instructions.

API Integrations and RAG

Because both Ollama and the llama-server expose an OpenAI-compatible API, you can use your local model as the brain for other applications. For example, if you program in Python, you can write scripts that use your local model to analyze data or search through documents via Retrieval-Augmented Generation. For a good understanding of these programming concepts, you can look at our guide on RAG for beginners.

6. Troubleshooting and Optimization

Working with local AI on Linux can sometimes lead to frustration, especially on older hardware. Here are the most common problems and their solutions:

• CUDA Out of Memory (OOM): If your model suddenly crashes while generating a long response, your VRAM is full. Solution: Choose a model with lower quantization (e.g., Q3 instead of Q5) via our guide on choosing a local model, or reduce the context size (-c 2048).
• System is painfully slow: If you run the LLM on the same PC you use for your Linux desktop (X11 or Wayland), the graphical interface competes with the LLM for VRAM. For maximum performance, boot the system directly to the command line (multi-user.target) and disable the desktop environment.
• Poor tokens-per-second (TPS): Check if the GPU offload is actually working. While generating text, you can run watch -n 1 nvidia-smi in a second terminal window. If the GPU usage (Volatile Uncorr. ECC) remains at 0%, the model is running entirely on your CPU. Re-evaluate the installation of your CUDA toolkit.

Conclusion

Linux is without a doubt the most powerful foundation for running local LLMs. Whether you choose the plug-and-play simplicity of Ollama via a simple systemd service, or the surgical precision of a self-compiled llama.cpp instance; you now have full control over your own, private AI without being dependent on cloud providers. Experiment with different models, monitor your VRAM usage, and discover the unprecedented possibilities of locally hosted AI.