Running an LLM on a Synology NAS: What works and what doesn't?

The popularity of local artificial intelligence has skyrocketed in recent years. More and more developers, tech enthusiasts, and small businesses want to be independent of large, commercial cloud providers like OpenAI or Google. If you already have a Synology NAS in your meter cupboard or office for your backups, media, and documents, it sounds like a logical next step: why not use that "always-on" server as your own local AI assistant?

Running a Large Language Model (LLM) on consumer hardware is surprisingly accessible these days, but a Network Attached Storage device is fundamentally designed differently than a high-end AI workstation. In this comprehensive guide, we will discuss exactly what is possible, where the hard limits of the hardware lie, and how you can get started hosting a language model on your own Synology NAS via Docker.

Why run an LLM on a NAS?

There are several compelling reasons to host a language model locally instead of using a cloud service. The main driver is undoubtedly data sovereignty and privacy. When you have a document summarized by a public cloud service, this data leaves your network. This is often undesirable or even prohibited when processing sensitive personal data, financial administration, or internal company documents.

In addition, the cost aspect plays a role. Commercial APIs charge fees per generated or read token. With frequent use, this can quickly add up. A local model, running on hardware you already own and that is already on 24 hours a day, has no variable subscription costs. For a fundamental understanding of the concepts behind these language models, the architecture, and how factors like parameters influence them, we recommend first reading the theoretical introductions on leren.llmnet.nl.

A NAS is designed as a robust, always-available server. By hosting a model on it, you create a centralized "AI brain" for your entire home network. Every device within your network—whether it is your laptop, smartphone, or tablet—can communicate with this centrally hosted model via a web interface or local API.

The harsh reality of hardware: CPU versus GPU

The biggest challenge in running an LLM on a NAS lies in the hardware architecture. Modern AI models are optimized for parallel processing. Graphics cards (GPUs) with their thousands of computing cores and extremely fast video memory (VRAM) are the absolute masters of this. However, a standard Synology NAS, such as the popular DS920+, DS923+, or DS1522+, does not have a dedicated GPU. They are equipped with energy-efficient processors, such as the Intel Celeron series (for example, the J4125) or AMD Ryzen embedded chips (such as the V1500B or R1600).

This means that all calculations for generating text (the inference) must run exclusively via the Central Processing Unit (CPU). A few years ago, this was virtually impossible, but thanks to software projects like llama.cpp, it is now very much possible to run models efficiently on a CPU. However, it remains significantly slower than GPU acceleration.

System Memory (RAM): The absolute bottleneck

Because your NAS has no video memory, the entire language model must be loaded into the regular system memory (RAM) during text generation. This is immediately the biggest limitation for Synology users. Many Plus models come standard with only 2 GB or 4 GB of RAM. That is more than enough for file sharing and lightweight Docker containers, but completely inadequate for an LLM.

Before you start this project, a RAM upgrade is almost inevitable. Although Synology often specifies conservative maximum RAM specifications (such as a maximum of 8 GB or 32 GB), in practice it is often possible to install (unofficial) modules, giving you 16 GB, 32 GB, or sometimes even 64 GB of RAM at your disposal. Always check if other users with your specific NAS model have performed successful upgrades.

Quantization: The technique that makes it possible

You may have noticed the term "quantized" or "4-bit" in the table above. If you were to download a model in its original, uncompressed form (often 16-bit precision), it would be far too large for a NAS's RAM, and the memory bandwidth would be insufficient. To solve this, we use quantization. This is a technique where the precision of the model's internal weights is reduced from complex floating-point numbers (float16) to simpler integers (such as 4-bit integers).

The most popular format for CPU inference is GGUF (formerly GGML). By using a GGUF model with Q4 (4-bit) quantization, you roughly halve the required amount of RAM and significantly increase generation speed, while the loss in intelligence and text quality is surprisingly minimal in practice.

Expectations and real-world performance

If you decide to install an LLM on your Synology NAS, managing expectations is crucial. You will not experience the lightning-fast real-time responses you are used to from paid cloud services. The speed of text generation is expressed in tokens per second (t/s), where one token roughly corresponds to three-quarters of a word.

As a rough estimate, with a current Synology featuring an AMD Ryzen V1500B or R1600 processor, you can expect about 2 to 5 tokens per second when running a 7B or 8B model. This means you will see the words appear on your screen at a calm, reading pace. For chat applications where you are waiting for an answer, this requires some patience. However, for asynchronous background tasks—such as scanning and summarizing incoming emails or extracting data from internal documents—this speed is more than sufficient.

Step-by-step: How to install an LLM via Docker?

The safest and most organized way to run a language model on your Synology NAS is by using Docker (in DSM 7.2 and newer, this is called Container Manager). For this, we use the software Ollama, a wonderful project that completely abstracts away the complexity of running local models behind a simple API and command-line interface.

Step 1: Prepare Container Manager

Make sure the Container Manager application is installed via your NAS's Package Center. Open File Station and navigate to the shared folder docker (this is created by default when installing Container Manager). Inside this folder, create a new subfolder named ollama. This is important because we will store the model files here. If you don't do this, you will lose all downloaded models as soon as you restart the NAS or the container.

Step 2: Install Ollama

You can install Ollama via the graphical interface of Container Manager, but it is often faster and more accurate to do this via a Docker Compose script (or Project in Container Manager). Create a new project and use the following configuration:

version: '3.8'
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - /volume1/docker/ollama:/root/.ollama

Note that you may need to adjust the path /volume1/docker/ollama if your docker folder is located on a different volume (for example, /volume2/). Start the project. The NAS will now download the necessary files and start Ollama. From that moment on, the service will listen on port 11434.

Step 3: Download a model

Now that Ollama is running, it is essentially still an empty shell; we still need to load a specific language model. Open an SSH connection to your NAS (SSH must be enabled in the Control Panel for this). Once you are logged in, open the terminal of your running Ollama container with the following command:

sudo docker exec -it ollama bash

Next, you can start a model of your choice. For NAS systems, we strongly recommend starting with an efficient, smaller model like Phi-3 or Qwen. You can find more information about the capabilities of specific lightweight architectures in our overview of open-source models. Type in the bash terminal:

ollama run phi3

Ollama will now automatically download the correct GGUF files for Phi-3 (this may take a while depending on your internet connection). Once the download is complete, you will see a prompt (>>>) and you can type your first message. Congratulations, you are now communicating directly with a neural network running entirely on your own NAS!

Step 4: Add a graphical interface

Command lines are fun for testing, but for daily use, you want a graphical interface similar to ChatGPT. The most popular choice for this is Open-WebUI. This is a separate Docker container that you can link to your running Ollama installation. You can add it to your existing Compose file or run it as a separate project. In the settings of Open-WebUI, you then enter the local IP address of your NAS (including port 11434) so that the web interface can communicate with the Ollama API backend.

Conclusion

The answer to the question of whether you can run a fully-fledged LLM locally on a Synology NAS is a resounding 'yes', but with some clear caveats. Without a powerful GPU, you will have to make compromises in terms of speed and model size. Do not expect real-time response times from giant 70B models, but instead enjoy the privacy, sovereignty, and educational process of self-hosting highly capable, quantized 4B to 8B models. With the right RAM upgrade and a correctly configured Docker environment, you can successfully transform a modest storage server into a local, private AI laboratory.