Running a local language model (LLM) in Docker: The complete guide

Running an LLM (Large Language Model) locally gives you unprecedented control over your privacy and data. No subscriptions, no data leaks to the cloud, just pure computing power on your own hardware. While you can install software like Ollama or LM Studio directly on your operating system, many developers and hobbyists choose to run their AI infrastructure in Docker.

Docker is a platform that allows you to package applications into isolated containers. Everything the language model needs to function (libraries, dependencies, and configurations) is contained in a single package. However, running heavy, hardware-intensive AI models in a container works fundamentally differently than hosting a simple web server. In this guide, we discuss why you would (or wouldn't) want to do this, how to handle hardware acceleration, and provide a practical blueprint for your own configuration.

Why run an LLM in Docker?

Choosing Docker brings several significant advantages, especially if you are building complex AI applications or want to keep your system clean.

Complete isolation (no 'Dependency Hell'): AI applications often require specific versions of Python, PyTorch, CUDA libraries, and other dependencies. By running everything in Docker, you prevent these packages from conflicting with software already on your computer.
Easy to move (Portability): Built a working setup on your laptop? You can transfer the exact same Docker container to a powerful server or a cloud environment with a single command. The environment is identical everywhere.
Reproducibility: Using a docker-compose.yml file, you define your entire architecture as code. You always know exactly which version of the AI model and which ports you are using.
Microservices: In practice, you often don't just run a model. You might want to add a vector database for contextual searches, or a graphical interface as described in our guide on setting up Open WebUI. Docker Compose effortlessly links these different containers together via an internal network.

Would you like to expand this setup later and train the model on your own company documents? Then we would like to refer you to the learning platform for our external course RAG for beginners.

The GPU Pitfall: Why Docker is sometimes slower

While Docker is fantastic for software isolation, it introduces a challenge regarding hardware. Language models require an extreme amount of computing power, preferably provided by a GPU (graphics card). And that is exactly where the catch lies: by default, a Docker container has no access to the host machine's hardware.

Without explicit configuration, an LLM in Docker runs exclusively on your processor (CPU). Because a CPU processes data sequentially rather than in parallel, text generation will be painfully slow (often less than 2 tokens per second). To use the GPU, it must be 'passed through' to the container. How well this works depends entirely on your operating system.

Differences per operating system

1. Linux with an NVIDIA GPU (The Ideal Situation)

Docker was originally built for Linux. On a Linux machine, containers directly share the kernel (the heart) of the operating system. If you install the NVIDIA Container Toolkit, you can pass the power of your graphics card to the container with virtually no performance loss. For serious local AI servers, this is by far the best choice. Also read our guide on hardware for local LLMs to see which cards are most suitable.

2. macOS (Apple Silicon: M1/M2/M3/M4)

On a Mac, Docker does not run natively. Docker Desktop for Mac creates a hidden Linux virtual machine (VM) in the background. The problem? macOS currently barely, if at all, allows passing the powerful integrated GPUs of Apple Silicon chips efficiently to this Linux VM. As a result, LLMs in Docker on a Mac often fall back to the CPU, leading to dramatically poor performance compared to native applications. Advice: On a Mac, it is better to use native software like Ollama or LM Studio outside of Docker, unless you accept a slow CPU fallback for testing purposes.

3. Network Attached Storage (NAS)

Running an LLM on a home server is popular, but an average NAS does not have a powerful graphics card. As a result, you are bound to CPU inference and limited RAM. This makes generating responses slow. It can be useful for asynchronous background tasks, but do not expect a real-time chat experience. You can read more about this in the article about an LLM on a Synology NAS.

Practical Guide: Setting up Docker for LLMs

If you have decided that Docker is the right path for you (for example, on a Linux machine or Windows with WSL2), there are four crucial concepts you must configure correctly: volumes, networks, memory limits, and updates.

1. Docker Volumes (Prevent losing your models)

LLMs are large files. A model like Llama 3 (8B) easily takes up 4 to 5 gigabytes. When a Docker container stops or is deleted, all data stored inside it disappears by default. Without so-called volumes, you would have to redownload the gigabytes of model files with every restart.

With a volume, you link a folder on your own hard drive (the host) to a folder in the container. Even if you delete the container to update it, your downloaded AI models remain safely stored on your drive.

2. Ports and Network

AI software in a container often runs a local API server (for example, on port 11434). To access this API from your own computer or from another application, you must 'map' the container's port to a port on your host machine. Without this mapping, the model remains isolated and cannot receive commands.

3. Setting memory limits

Language models consume RAM like water. If a model tries to use more memory during generation than your system has available, the operating system steps in and terminates random processes (the dreaded OOM killer, Out Of Memory). Within Docker, you can set hard limits (mem_limit) so that the container can never crash your entire system.

4. Updating without data loss

One of the best aspects of Docker is maintenance. Is there a new version of the LLM software? Because you have safely placed your model files in a volume, an update consists of just three simple steps in the terminal:

docker pull ollama/ollama:latest (Download the latest software version)
docker stop [container_name] && docker rm [container_name] (Remove the old container)
Restart the container with your existing volume command.

Your software is immediately updated, and your models are still there.

Step-by-step example: Ollama via Docker Compose

The easiest way to bring all of this together is via Docker Compose. With a single configuration file, you start both the engine that runs the model (Ollama) and a graphical interface. Here is a production-ready docker-compose.yml example.

Note: This example assumes you are using Linux with the NVIDIA Container Toolkit installed. Without this toolkit, Docker will ignore the deploy section for the GPU or throw an error.

version: '3.8'

services:
  # The AI Engine (Ollama)
  ollama:
    image: ollama/ollama:latest
    container_name: local-llm
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    # Link the NVIDIA GPU to the container
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  # The Graphical Interface (Open WebUI)
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: llm-interface
    restart: unless-stopped
    ports:
      - "3000:8080"
    volumes:
      - open-webui_data:/app/backend/data
    environment:
      # Tell the UI where to find the AI Engine in the Docker network
      - OLLAMA_BASE_URL=http://ollama:11434
    depends_on:
      - ollama

# Define the volumes for persistent data
volumes:
  ollama_data:
  open-webui_data:

With this file, you only need to navigate to the relevant folder in your terminal and run the command docker-compose up -d. Docker downloads the required software, mounts the folders, connects your GPU, and starts an internal network. You can then open your web browser at http://localhost:3000 to chat with your own local model.

Conclusion

Running a local LLM in Docker offers a clean, reproducible, and secure way to experiment with artificial intelligence. While it is the gold standard for Windows and Linux users with an NVIDIA graphics card thanks to excellent GPU passthrough, macOS users must be cautious of performance loss due to virtualization. Always make sure to work with volumes to preserve your large model files and be aware of the memory consumption of your applications.