Configuring power limits and fan curves for GPU servers

Anyone setting up their own machine for local language models is in the hardware management and operational optimization phase: the step immediately following component selection and base software installation. For the initial component choice, consult the foundational overview on what hardware is needed for local LLMs to see which cards and memory bandwidths are suitable for inference tasks. Once the drivers are installed on a Linux system, as detailed step-by-step in the guide on running local LLMs on Linux, practical experience shows that default factory settings are rarely optimal for continuous AI inference, server processes, or prolonged batch processing.

Standard consumer and workstation GPUs are tuned out of the box to boost aggressively toward the highest clock speeds until reaching their maximum thermal or electrical limits. During token generation, however, that final ten to twenty percent of electrical power yields virtually no perceptible speedup, while heat output and noise levels increase exponentially. In this guide, we walk through how configuring power limits and custom fan curves ensures a whisper-quiet, thermally stable, and energy-efficient server environment.

Why factory settings are suboptimal for AI inference

Modern GPUs dynamically scale their internal clock frequencies using algorithms such as NVIDIA GPU Boost. These mechanisms linearly increase voltage and core clock speeds until hitting the preset Thermal Design Power (TDP) ceiling. Under default settings, a modern graphics card with a factory TDP of 350 watts or 450 watts continuously consumes this maximum capacity during heavy prompt processing (prefill) and token generation.

For language models, however, the throughput during the autoregressive generation phase is primarily memory-bandwidth-bound by the VRAM, and far less dependent on the raw compute power of the CUDA or Tensor cores. After all, for every generated token, all model weights must be transferred anew from video memory to the compute units. This creates a law of diminishing returns: increasing power from 280 watts to 450 watts often yields only a marginal gain in token speed, while power consumption rises by more than sixty percent. The theoretical foundation of this cost structure is detailed in the analysis of the electricity costs of running local AI.

In addition, unconstrained heat generation leads to thermal degradation and sudden thermal throttling. When the GPU core temperature approaches 83 degrees Celsius or the VRAM exceeds 95 to 105 degrees, the BIOS intervenes by aggressively scaling down clock speeds. This causes erratic response times, increased latency spikes, and unnecessary mechanical wear on fan ball bearings and PCB components in a 24/7 server environment.

NVIDIA Persistence Mode and power limits via nvidia-smi

To apply changes directly and reliably via the command-line interface, the tool nvidia-smi serves as the primary entry point. By default, the Linux driver initializes the GPU state only when a process opens an active CUDA context, and immediately tears down this state as soon as the process terminates. Consequently, dynamically configured power limits are instantly lost after each standalone task or API call. Enabling Persistence Mode is therefore an absolute prerequisite.

# Activeer persistence mode voor alle aanwezige GPU's
sudo nvidia-smi -pm 1

# Controleer de huidige, minimale en maximale vermogenslimieten
nvidia-smi -q -d POWER

The output of the query command above reveals three critical values: Min Power Limit, Max Power Limit and Default Power Limit. A high-end 450W card, for example, typically indicates a minimum of 150 watts, a default of 450 watts, and a hardware-enforced maximum of 600 watts. Once persistence is enabled, a specific power limit in watts can be defined for each card using the flag -pl (power limit). In a multi-GPU configuration, a card-specific index can be targeted using the parameter -i.

# Stel het vermogenslimiet van GPU 0 in op 280 Watt
sudo nvidia-smi -i 0 -pl 280

# Stel het vermogenslimiet van GPU 1 in op 260 Watt
sudo nvidia-smi -i 1 -pl 260

This setting takes effect immediately at the hardware level. The voltage regulator on the PCB directly adjusts the maximum voltage supply, allowing the core temperature to quickly stabilize at a significantly lower level without requiring a restart of either the model or the server.

The thermodynamic relationship between wattage, temperature, and token throughput

Determining the ideal wattage for a specific server requires understanding the interplay between compute intensity, memory saturation, and thermal dissipation. During initial prompt processing (the prefill phase), input tokens are processed in parallel via dense matrix multiplications (GEMM), driving Tensor cores to maximum utilization. In this brief phase, higher wattage has a measurably positive impact on processing times for large context windows.

During the autoregressive decode phase (generating text token by token), however, the bottleneck shifts entirely to the memory bus. The compute units are constantly waiting for weights to be loaded from the GDDR6(X) memory. Supplying extra power to the GPU core during this phase primarily increases temperatures and leakage current losses (Joule heating), without proportionally increasing memory bandwidth. Anyone looking to evaluate how these dynamics relate to total system power consumption will find the appropriate measurement methodology in the article on measuring power consumption in a local LLM setup.

The calculation model below illustrates the theoretical and qualitative progression of power efficiency at various levels of power limiting on a high-end GPU with a default power rating of 450 watts:

For continuous inference servers and interactive API endpoints, the optimal operational sweet spot typically lies between 60% and 75% of the default power limit. This keeps token generation speeds virtually indistinguishable from stock settings, while dramatically reducing overall heat generation.

Making settings persistent with a systemd service

Commands entered manually via nvidia-smi do not persist across an operating system reboot. To ensure that the server immediately reboots into its optimized state following a scheduled restart, power outage, or automatic kernel update, we set up a dedicated systemd unit.

Create a new service file in systemd's central configuration directory:

# /etc/systemd/system/nvidia-power-limit.service
[Unit]
Description=NVIDIA GPU Power Limits en Persistence Mode
After=syslog.target network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/nvidia-smi -pm 1
ExecStart=/usr/bin/nvidia-smi -pl 280

[Install]
WantedBy=multi-user.target

If the system contains multiple distinct graphics cards, multiple ExecStartlines can be placed sequentially, assigning a specific wattage per GPU index:

# Multi-GPU variant voor gemengde kaarten
ExecStart=/usr/bin/nvidia-smi -i 0 -pl 280
ExecStart=/usr/bin/nvidia-smi -i 1 -pl 240

Then enable and start the service via Linux's service manager:

sudo systemctl daemon-reload
sudo systemctl enable nvidia-power-limit.service
sudo systemctl start nvidia-power-limit.service

# Controleer of de service succesvol is uitgevoerd
systemctl status nvidia-power-limit.service

Managing fan curves on headless Linux servers

On desktop installations with a graphical desktop environment such as GNOME or KDE, fan curves can be easily adjusted using GUI tools like GreenWithEnvy. On a headless production server, however, an active display server is absent. The official NVIDIA Linux drivers have traditionally required interaction with an active X11 server to allow fan control via the so-called Coolbits flag.

To enable manual fan control on a server without a physically connected monitor, we configure a virtual X server (dummy X server) with Coolbits support:

# Genereer een minimale xorg.conf met Coolbits (bitmask 28 voor klok- en fancontrole)
sudo nvidia-xconfig --cool-bits=28 --allow-empty-initial-configuration --enable-all-gpus

Next, a virtual Xorg session can be started in the background, after which the fan curve can be controlled using nvidia-settings:

# Start X op display :0 in de achtergrond
sudo Xorg :0 &

# Schakel handmatige ventilatorsturing in voor GPU 0
DISPLAY=:0 XAUTHORITY=/var/run/lightdm/root/:0 nvidia-settings -a "[gpu:0]/GPUFanControlState=1"

# Zet de ventilatorsnelheid vast op een constant percentage van 65%
DISPLAY=:0 XAUTHORITY=/var/run/lightdm/root/:0 nvidia-settings -a "[fan:0]/GPUTargetFanSpeed=65"

For modern headless environments where running an Xorg daemon is considered unwanted overhead, lightweight alternatives are available such as nvfancontrol or scripts that communicate directly with the NVML (NVIDIA Management Library) C-API. These daemons run entirely within user space or as a native service and do not require an X11 architecture.

Thermal Dynamics and Asymmetric Load in Multi-GPU Setups

When two or more graphics cards are placed directly next to each other in PCIe slots, a significant asymmetric thermal load occurs. The top card typically pulls in the preheated exhaust air from the bottom card, causing the temperature of the top GPU to run ten to fifteen degrees higher at an identically set wattage. For those combining multiple cards in a single chassis, in-depth instructions on PCIe slot configurations and bandwidth allocation can be found in the guide on combining two GPUs for larger local language models.

In such a setup, applying identical power limits is a classic design flaw. A much more robust strategy consists of differentiated power limits:

During heavy inference with production frameworks such as vLLM or Ollama, this balanced temperature distribution ensures consistent, predictable latency without thermal throttling spikes. More on the behavior and performance of these software stacks under continuous load can be found in the review of vLLM versus Ollama for self-hosted production.

Automation and Dynamic Monitoring with Python and NVML

For advanced management, a custom Python background daemon can be set up that uses the official NVML bindings (pynvml). The major advantage of this is that the control logic can respond not only to the chip's core temperature, but also to the VRAM temperature (Memory Junction Temperature). During intensive quantization and tensor operations, the video memory can become hot while the GPU core itself remains relatively cool.

import time
import subprocess
from pynvml import *

nvmlInit()
device_count = nvmlDeviceGetCount()

# Temperatuurgrenzen voor dynamische ventilatorkromme
FAN_CURVE = [
    (45, 35),  # <= 45°C -> 35% fanspeed
    (55, 50),  # 55°C    -> 50% fanspeed
    (65, 70),  # 65°C    -> 70% fanspeed
    (75, 85),  # 75°C    -> 85% fanspeed
    (82, 100)  # >= 82°C -> 100% fanspeed
]

def bereken_fanspeed(temp):
  for t_grens, speed in FAN_CURVE:
    if temp <= t_grens:
      return speed
  return 100

def monitor_en_stuur():
  for i in range(device_count):
    handle = nvmlDeviceGetHandleByIndex(i)
    temp_core = nvmlDeviceGetTemperature(handle, NVML_TEMPERATURE_GPU)
    power_mw = nvmlDeviceGetPowerUsage(handle)
    power_w = power_mw / 1000.0
    
    doel_speed = bereken_fanspeed(temp_core)
    print(f"GPU {i}: Kern={temp_core}°C | Verbruik={power_w:.1f}W | Fan Doel={doel_speed}%")

if __name__ == "__main__":
  try:
    print("NVML Monitor Daemon gestart...")
    while True:
      monitor_en_stuur()
      time.sleep(3)
  except KeyboardInterrupt:
    print("Afsluiten...")
  finally:
    nvmlShutdown()

Such scripts can also incorporate a safety loop: if the core temperature unexpectedly exceeds a critical threshold, the script can use a subprocess call to nvidia-smi -pl to automatically lower the power limit temporarily by an additional 30 watts until the components have cooled down.

Pitfalls, Hardware Limitations, and Practical Risks

Although setting power limits and custom fan curves is generally very safe, there are several important edge cases and hardware limitations that must be taken into account:

1. Minimum power thresholds: Every graphics card has a hard lower limit (Min Power Limit) built into its firmware. It is not possible to throttle a 450W card down to 50 watts via nvidia-smi ; the driver will reject values below the minimum (often around 150W to 200W). Those who desire even lower power consumption during idle times must rely on the motherboard's automatic C-states and PCIe ASPM (Active State Power Management) power-saving profiles.

2. VRAM Junction vs. Core Temperature: Default fan curves in consumer card BIOSes look exclusively at the GPU core temperature. During long prompting sessions, the GDDR6X memory on the rear of a circuit board can become significantly hotter than the core. Therefore, always ensure adequate active airflow over the backplate of the graphics card.

3. Acoustic wear and hysteresis: Fans that constantly switch back and forth between 40% and 80% wear out significantly faster due to alternating mechanical torsional forces. Always build a hysteresis of at least 3 to 5 degrees into custom-written software, so the fan speed does not oscillate with minimal temperature fluctuations.

Step-by-step guide for a reliable production configuration

Structurally optimizing GPU servers for AI inference can be summarized into a clear step-by-step plan:

Step 1: Enable Persistence Mode. Ensure the GPU driver retains its state with sudo nvidia-smi -pm 1 to prevent losing configuration settings between API calls.

Step 2: Determine the power limit. Step down the wattage gradually (to around 60% to 75% of the factory TDP) and check whether the token generation speed remains acceptable for the specific workload.

Step 3: Persist via systemd. Write the settings into a /etc/systemd/system/nvidia-power-limit.service unit so that the policy survives system reboots.

Step 4: Monitor active cooling. Set asymmetric power limits on multi-GPU machines and ensure adequate fan curves via Xorg-Coolbits or an NVML daemon to prevent the video memory from overheating.

With this approach, a local AI server runs quieter, cooler, and significantly more efficiently, maximizing hardware lifespan while maintaining stable response times for all connected users.