Installing Text Generation WebUI on a Linux server

Along the route of running locally — from choosing components to hosting your own models — this guide sits squarely in the installation phase at server level. If you are looking for an overview of the physical system requirements, you can first consult what hardware local LLMs require to check whether the server has enough compute power. This guide builds specifically on the foundations from the overview article on running local LLMs on Linux and zooms in on installing and configuring Text Generation WebUI, better known in the open-source community as oobabooga.

Why Text Generation WebUI on a Linux server?

Text Generation WebUI sets itself apart from simpler interfaces such as Ollama through its modular design and unmatched parameter control. Where consumer solutions abstract many settings behind a black box, this platform exposes the full inference stack. You get direct control over loaders such as llama.cpp, ExLlamaV2, Transformers, and AutoAWQ within a single central interface. That makes the platform particularly well suited to Linux servers where different model formats are evaluated side by side.

That flexibility does come with clear trade-offs. The platform is designed as an experimentation and evaluation environment for one administrator or researcher at a time. It lacks built-in multi-user account management and advanced scheduling with continuous batching for simultaneous users. If all you want is a user-friendly chat interface for several users on an office network, you are better off setting up Open WebUI in combination with a specialized backend. You choose Text Generation WebUI when you want deep control over context size, layer offloading, prompt templates, samplers, and LoRA adapters without compromise.

Preparing system requirements and drivers

For stable operation on a Linux server, the right NVIDIA drivers and a clean software environment are essential. Although Text Generation WebUI isolates its own Python packages and CUDA runtimes in an internal Conda environment, the Linux host system needs a functioning NVIDIA driver compatible with the graphics card.

First check whether the NVIDIA GPU is recognized correctly by the kernel and which driver is currently active, from the terminal:

nvidia-smi

If this command is missing or returns an error, install the official headless driver through the Linux distribution's package manager. First update the package sources and install the required build tools and driver modules:

sudo apt update && sudo apt install -y build-essential dkms git curl wget
sudo ubuntu-drivers install --gpgpu
sudo reboot

After the restart, nvidia-smi should show a status overview with the GPU name, the driver version, and VRAM usage. Also make sure there is enough disk space available. Besides the roughly 8 to 12 GB for the application and its Python dependencies, modern language models require considerable storage on a fast NVMe SSD.

Installing through the standalone script

The developers of Text Generation WebUI provide an automated installation script that sets up an isolated Miniconda environment, pulls in PyTorch with the correct CUDA version, and compiles all dependencies. This avoids conflicts with system-wide Python packages.

Clone the official repository into a target directory of your choice (under /opt or in the home directory of a service user, for example) and run the start script:

cd /opt
sudo git clone https://github.com/oobabooga/text-generation-webui.git
sudo chown -R $USER:$USER /opt/text-generation-webui
cd /opt/text-generation-webui
./start_linux.sh

During the first run, the script asks a number of questions in the terminal:

1. GPU selection: Choose option A) NVIDIA GPU to enable support for CUDA acceleration.
2. CUDA version: For recent graphics cards, choose the default recommended CUDA 12.x compiler.
3. Model download: The script asks whether you want to download a model right away; this can be skipped by pressing Enter, since we will configure it more precisely later through the interface or the CLI.

The script now downloads the required runtime files. Depending on the server's connection speed, this process takes a few minutes. Once installation is complete, the WebUI starts on port 7860 on the loopback address (127.0.0.1).

CLI parameters and network configuration for server use

By default, Text Generation WebUI listens only for local connections from the machine itself. On a headless Linux server without a graphical desktop environment, the interface is therefore not directly reachable from other workstations on the network. To grant access, specific startup flags have to be supplied.

Create or edit the configuration file CMD_FLAGS.txt in the application's root directory. In it you define arguments that are loaded automatically at every startup:

nano CMD_FLAGS.txt

Add the desired parameters there to make the web interface listen on all network interfaces, activate the OpenAI-compatible API, and set up basic security:

--listen
--listen-port 7860
--api
--api-port 5000
--gradio-auth beheerder:GeheimWachtwoordHier!

Choosing model loaders: llama.cpp versus ExLlamaV2

In the Model tab of the WebUI, you choose which engine handles inference. The choice of the right model loader is directly tied to the file format of the model you are deploying. Consult the article on quantization and the trade-off between file size and quality to determine which format suits your specific memory budget.

When using the llama.cpp loader with a GGUF model, you set the parameter n-gpu-layers . This determines how many layers of the neural network are loaded into fast GPU memory. If the server has enough VRAM for the chosen model, set this value to the maximum number of layers so the entire model runs on the GPU for optimal performance.

If you choose ExLlamaV2 with an EXL2 model, set the gpu-split and any cache_8bit or cache_4bit options. Compressing the KV cache to 4-bit makes the context space consume substantially less VRAM, which lets you stay within the memory budget of a single graphics card even with larger context windows.

Production deployment with systemd and process management

To make sure Text Generation WebUI starts automatically when the server reboots and recovers from any crashes, we configure a systemd service. Create a new service file:

sudo nano /etc/systemd/system/oobabooga.service

Place the service definition below in the file. Adjust the paths and the username to match your configuration:

[Unit]
Description=Text Generation WebUI (oobabooga) Service
After=network.target nvidia-persistenced.service

[Service]
Type=simple
User=beheerder
WorkingDirectory=/opt/text-generation-webui
ExecStart=/bin/bash /opt/text-generation-webui/start_linux.sh
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
Environment="PYTHONUNBUFFERED=1"

[Install]
WantedBy=multi-user.target

Then reload the configuration, enable the service for automatic startup at system boot, and start the process immediately:

sudo systemctl daemon-reload
sudo systemctl enable oobabooga.service
sudo systemctl start oobabooga.service

You can easily follow the status and live logs of the running service with journalctl:

sudo journalctl -u oobabooga.service -f

Using the OpenAI-compatible API

Once the parameter --api is active in CMD_FLAGS.txt, the server listens by default on port 5000 for REST calls that follow the OpenAI API specification. That lets you connect external applications, scripts, or dashboards directly.

The endpoint is particularly powerful when you want to use the local server for advanced search systems. Read more about how a language model can enrich documents in the explanation of RAG for beginners and making your own files searchable to discover how to connect the API to a document pipeline.

Test the connection from another Linux workstation on the local network with curl:

curl http://SERVER-IP:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "instruct-model",
    "messages": [
      {"role": "system", "content": "Je bent een behulpzame assistent."},
      {"role": "user", "content": "Leg in drie zinnen uit wat Linux is."}
    ],
    "temperature": 0.7,
    "max_tokens": 150
  }'

Tuning language behavior and parameters for Dutch input

To evaluate how the server performs on real tasks, you can use a prompt that requires both contextual logic and correct sentence structure. For practical tips on optimizing system prompts and model behavior in Dutch, consult the guide on getting your AI to perform better in Dutch.

Example prompt for business tasks:
"Write a concise three-paragraph policy proposal on storing company data internally within the frameworks of the GDPR. Use a businesslike, formal tone."

Example of typical model output:

"This policy proposal describes the guidelines for the storage and processing of personal data within the internal infrastructure, in accordance with the General Data Protection Regulation (GDPR). The aim is to safeguard the confidentiality and integrity of company and customer data by conducting data processing exclusively on local servers, without the involvement of external cloud providers."

"All departments working with sensitive data must store it within walled-off network segments. Access is granted solely on the basis of the 'need-to-know' principle and is secured with multi-factor authentication. Periodic audits monitor compliance with these access rights and document every data mutation in a central log."

"By keeping full control over data storage in-house, the organization eliminates the risk of unauthorized transfer to third countries. This proposal lays the foundation for a robust compliance framework that is reviewed periodically in response to current legislation and regulations."

Points of attention when measuring performance:

• Speed differences per loader: Dedicated GPU loaders such as ExLlamaV2 generally achieve considerably higher token throughput when the model fits entirely in VRAM than flexible CPU/GPU hybrid loaders such as llama.cpp.
• Latency and TTFT: The response time to the first token (time to first token) is largely determined by the prompt evaluation speed of the chosen backend and the length of the context.
• Memory distribution: Account for both the base memory for the model weights and the dynamic VRAM allocation that grows as the active context window expands.

Privacy and data security

The main argument for a dedicated Linux server running Text Generation WebUI is absolute control over data flows. When the application is configured correctly, no prompt, generated output, or document fragment leaves the local network. For a full compliance overview, also see the guide to privacy-friendly AI use to mitigate operational risks.

There are, however, two important points where data can travel across the network:

• Model downloads: If you download models through the Model (with the built-in Hugging Face downloader), the server connects to Hugging Face's external storage servers. This happens only while the weights are being downloaded.
• External tunneling: In a production or server environment, do not use external proxy or sharing options (such as --share), unless network traffic is explicitly routed through your own VPN or a walled-off tunnel.

Power consumption and hardware load

A Linux server that stays operational continuously as an LLM backend has a variable power profile. Actual consumption depends heavily on the state of the GPU: idle or actively computing. For methods to map consumption accurately, read the article on what the power consumption of local AI costs.

Illustrative calculation: Suppose a server draws an average of 70 watts at rest and peaks at 350 watts during active generation. In a hypothetical scenario of 22 hours at rest and 2 hours of continuous generation per day, daily consumption comes to roughly 2.24 kWh. At a hypothetical rate of € 0.30 per kWh, that results in an indicative figure of about € 20 per month. This is a purely theoretical example calculation; actual values depend on the specific power supply efficiency, the number of GPU cores, ambient temperature, and dynamic energy tariffs.

To limit unnecessary consumption, the NVIDIA Persistence Daemon can be enabled, while setting a power limit with nvidia-smi -pl can help optimize energy efficiency per generated token.

Troubleshooting server use

If installation or model loading unexpectedly stalls, consult the overview of troubleshooting out-of-memory messages and slow tokens for in-depth diagnostic steps. The most common situations with Text Generation WebUI on Linux are:

1. CUDA out of memory (OOM): The model or the requested context window exceeds available VRAM. In the Model tab, lower the parameter n-gpu-layers (with llama.cpp) or activate a compressed cache_4bit (with ExLlamaV2).
2. Interface unreachable from other machines: Check whether the flag --listen is actually present in CMD_FLAGS.txt and check the Linux firewall with sudo ufw status. Open port 7860 if needed with sudo ufw allow 7860/tcp.
3. Unexpectedly slow generation: This generally indicates that the model has been partly loaded into slower system RAM, or that the required CUDA acceleration is not being addressed correctly, causing computations to run on the CPU.

Conclusion and next steps

Text Generation WebUI gives administrators and advanced users a high degree of control over local inference on a Linux server. With support for a range of loaders and model formats, combined with an OpenAI-compatible API and process management through systemd, it forms a flexible foundation for both evaluation work and local integrations.

Once the base installation is running, the server can be extended further with walled-off network access, automated pipelines, or specific fine-tuning adapters to deploy models even more precisely within your own infrastructure.