Combining Two GPUs for Larger Local Language Models

In the continuous journey of local AI — from selecting basic components and software installation to scaling up to a full-fledged home server — the physical VRAM limit of a single graphics card often represents the first major hurdle. Anyone looking to scale up from compact 7B or 8B models to advanced 32B or 70B architectures quickly discovers that a single consumer card with 12 or 16 GB of video memory falls short. Before considering the jump to extremely expensive enterprise hardware, combining two graphics processors (GPUs) offers a flexible and modular approach to doubling total VRAM.

To understand the hardware requirements of local models and see how VRAM compares to system memory, you can start with the overview of what hardware is needed for local LLMs. In this guide, we look specifically at the technical inner workings of dual-GPU setups: the fundamental difference between sequentially distributing compute layers and splitting matrices in parallel, the impact of the motherboard and PCIe lanes on throughput, and the practical configuration of inference engines such as llama.cpp, Ollama, and vLLM under Linux. For setting up the underlying distribution, consult the guide on running local LLMs on Linux.

Why Two GPUs: Mathematics of the VRAM Wall

When running autoregressive language models, all model parameters must reside continuously in fast, addressable memory. When a model contains 70 billion parameters and is quantized to 4-bit precision, the static model weights alone require 38 to 40 GB of storage. Additionally, the KV cache (the key-value memory that retains previous tokens within the context window) demands extra VRAM: at a context length of 8,192 tokens and multiple concurrent sequences, this adds another 2 to 6 GB, on top of fixed CUDA runtime allocations.

A single consumer graphics card with 16 GB or 24 GB cannot fit a 70B model with a usable context length. As soon as a portion of the network layers is offloaded to the CPU's standard DDR4 or DDR5 system memory, throughput drops drastically. This performance loss occurs because system memory bandwidth (around 50 to 80 GB/s) is many times lower than the internal bandwidth of video memory (around 900 to 1000 GB/s on GDDR6X).

Pairing two cards with 24 GB VRAM each creates a theoretical pool of 48 GB. This allows the entire model, including a generous context buffer, to remain within the fast video memory, drastically reducing processing time per token compared to CPU-memory hybrid setups.

The Two Flavors of Parallelization: Pipeline vs. Tensor

Distributing a neural network across multiple GPUs can roughly be done using two strategies: layer-by-layer splitting (Pipeline Parallelism) or concurrent matrix splitting (Tensor Parallelism). This distinction determines which hardware requirements and software engines apply.

With Pipeline Parallelism, as used in llama.cpp, the chain functions like a relay race. GPU 0 computes the first set of layers and transmits the resulting activation vector (a few dozen kilobytes) across the PCIe bus to GPU 1. Because this data transfer occurs only once per token, the limited throughput of a standard motherboard has virtually no negative impact on generation speed.

With Tensor Parallelism , the weight matrices within each attention and feedforward layer are divided. Both chips perform computations in parallel and must synchronize intermediate results after every layer. Those looking to build a scalable multi-user environment will find the implementation steps for this architecture in the guide on configuring vLLM for high throughput on Linux.

Hardware Requirements: Motherboard, PCIe Lanes, and Power Supply

Adding a second high-performance graphics card imposes specific requirements on the host machine's mechanical layout, communication lanes, and power supply.

PCIe Lane Distribution and Motherboard Selection

Standard desktop processors have a limited number of direct PCIe lanes from the CPU (typically 16 to 24 lanes). When two physical x16 slots are utilized, the motherboard usually splits these connections into an x8/x8 configuration via PCIe bifurcation. Budget motherboards sometimes route the second slot through the chipset at x4 speeds, introducing additional latency.

For Pipeline Parallelism, an x8/x8 setup on PCIe 4.0 or even PCIe 3.0 is more than sufficient; the measurable impact on sequential token generation is negligible. With Tensor Parallelism without NVLink, on the other hand, an x4 connection via a chipset results in noticeable slowdowns during the frequent All-Reduce communication phases.

Physical Space and Thermal Management

Consumer graphics cards with bulky coolers often occupy 3 to 3.5 expansion slots. Mounting two cards directly adjacent to each other restricts airflow to the top card. A motherboard with at least three slots of physical spacing, a PC case with targeted intake fans, or the use of shielded PCIe 4.0 riser cables is essential to prevent overheating and thermal throttling.

Power Supply (PSU) and Transient Power Spikes

High-end GPUs with a thermal design power of 300 to 450 watts each can experience brief surges during peak loads (transient spikes). Combined with the processor and peripherals, a dual-GPU workstation demands a high-quality 1200 to 1500 watt power supply with dedicated power cables per card, preferably compliant with the ATX 3.0 standard.

Combining Two Different GPUs: Possibilities and Limitations

Pairing two non-identical cards — such as a model with 24 GB VRAM and a secondary card with 12 or 16 GB VRAM — is a widely discussed scenario. Feasibility depends directly on the parallelization technique:

• Pipeline Parallelism (llama.cpp / Ollama): This fully supports asymmetrical hardware. Layers can be allocated proportionally (for example, 45 layers on a 24 GB GPU and 20 layers on a 12 GB GPU). The overall throughput, however, is bottlenecked by the slowest card in the chain, as the faster chip must wait for the preceding or succeeding set of layers to finish computing.
• Tensor Parallelism (vLLM / TensorRT-LLM): This requires symmetry. Matrices are split into equal chunks across the number of GPUs (tensor_parallel_size). If a 24 GB and a 16 GB card are combined, the engine can allocate at most 16 GB on each card, leaving 8 GB on the primary card unutilized. Furthermore, discrepancies in compute capability cause synchronization wait times after every transformer layer.

Practical Configuration: llama.cpp and Ollama

Multi-CUDA device support is directly integrated into the llama.cpp ecosystem. Using specific startup parameters, the allocation across GPU memories can be precisely tuned.

Using the flags -sm (split-mode) and -ts (tensor-split), the administrator controls how the compute layers are distributed across the available devices.

Manual Layer Allocation in llama-server

In the configuration example below, 80 transformer layers of a 70B model are evenly distributed across two identical 24 GB cards in a 50/50 ratio:

# Start llama-server met 80 lagen verdeeld over GPU 0 en GPU 1 (symmetrische 50/50 split)
./llama-server \
  -m /pad/naar/Llama-3.1-70B-Instruct-Q4_K_M.gguf \
  -ngl 80 \
  -sm layer \
  -ts 24,24 \
  -c 8192 \
  --host 0.0.0.0 \
  --port 8080

In an asymmetric configuration — for example, a 24 GB card in position 0 and a 16 GB card in position 1 — the ratio can be configured as a percentage:

# Asymmetrische toewijzing: 60% van het gewicht naar GPU 0, 40% naar GPU 1
./llama-server \
  -m /pad/naar/Qwen-2.5-32B-Instruct-Q4_K_M.gguf \
  -ngl 64 \
  -sm layer \
  -ts 60,40 \
  -c 16384

Management in Ollama

By default, Ollama detects all available NVIDIA graphics cards and automatically distributes the model based on free VRAM capacity. Using the environment variable CUDA_VISIBLE_DEVICES the selection of specific cards can be forced:

# Controleer eerst de beschikbare GPU-indices en geheugencapaciteit
nvidia-smi --query-gpu=index,name,memory.total --format=csv

# Start de Ollama-service expliciet met GPU 0 en GPU 1
CUDA_VISIBLE_DEVICES=0,1 ollama serve

Configuration in vLLM for Production Throughput

For scenarios requiring the processing of concurrent requests from multiple users, vLLM delivers optimized performance using PagedAttention and Tensor Parallelism. Calculations within individual layers are executed simultaneously.

Starting an OpenAI-compatible API server with a 70B AWQ-quantized model across two GPUs requires the parameter --tensor-parallel-size 2 (or -tp 2):

# Start vLLM met Tensor Parallelism over twee identieke GPU's
python3 -m vllm.entrypoints.openai.api_server \
  --model casperhansen/llama-3.1-70b-instruct-awq \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 8192 \
  --port 8000

During startup, vLLM reserves an identical amount of VRAM for weights on both cards and splits the dynamic KV cache evenly. Those setting up such a server within a broader network environment can consult the guidelines on how to prepare IT infrastructure for local LLMs regarding network security and server rooms.

Performance and Throughput Models: Single GPU vs. Dual GPU

Adding a second graphics card doubles total memory capacity, but does not necessarily double the generation speed for an individual interactive session. The table below shows an illustrative calculation example of how memory allocation and theoretical processing speeds compare across different model sizes and parallelization methods.

Three key dynamics emerge from this theoretical modeling:

• For compact models that already fit entirely within the memory of a single card, a second GPU provides no speed advantage. In fact, the necessary inter-GPU communication introduces minimal overhead.
• For models that exceed the capacity of a single card (such as 32B and 70B), a dual-GPU setup prevents layers from having to be offloaded to slow system RAM. This avoids the notorious performance collapse during memory transfers.
• Tensor Parallelism distributes the computational workload across both GPU cores simultaneously, which—given sufficiently fast interconnects—can result in higher prompt processing throughput than sequential layer splitting.

Quality on Dutch tasks with 70B models

Being able to run a 70-billion-parameter model operationally delivers significant advantages for complex text processing. Where smaller models (such as 7B or 8B) more frequently drop the ball on sentence structure or nuance during complex reasoning or long contexts, 70B models retain their logical coherence and stylistic consistency substantially better.

Particularly in formal Dutch correspondence, policy summaries, or legal documents, a larger model reduces the risk of anglicisms and contextual derailments. You can read about how to formulate optimal prompt structures for Dutch-language processing in the article on making your AI perform better in Dutch.

Power consumption and operational costs

Placing a simultaneous load on two graphics cards has a direct impact on power consumption. At idle, two modern high-end graphics cards collectively consume roughly 25 to 40 watts. During intensive inference workloads, the combined power draw of the GPUs can climb to 500 to 700 watts.

Because LLM inference is primarily bounded by memory bandwidth rather than the maximum clock frequency of the compute cores, lowering the power limit via the driver can significantly reduce power consumption without a noticeable loss in token throughput:

# Verlaag de vermogenslimiet van GPU 0 en GPU 1 naar bijvoorbeeld 280 Watt
sudo nvidia-smi -i 0 -pl 280
sudo nvidia-smi -i 1 -pl 280

A comprehensive economic evaluation covering kilowatt-hour rates and depreciation is described in the analysis on what local AI power consumption costs.

Privacy and data sovereignty in a local setup

The primary motivation for investing in a multi-GPU setup is complete data sovereignty. By hosting powerful 70B models locally, confidential business documents, medical records, or personal data can be processed without reliance on external cloud APIs or data transfers to third-party servers.

During the inference process, all prompts and responses remain within the local memory of the host machine. Guidelines for ensuring privacy and data security within local infrastructure are detailed in the overview on using AI in a privacy-friendly way.

Common pitfalls and troubleshooting

When setting up multi-GPU systems, specific configuration errors frequently occur. The three most common bottlenecks and their solutions are:

1. Unbalanced memory allocation (OOM on GPU 0)

This happens when software places all context buffers and initial allocations on the primary graphics card by default, causing GPU 0 to fill up while GPU 1 is barely utilized. In llama.cpp, this can be corrected by setting the ratio in the parameter -ts manually adjusting it, for example to -ts 22,24, to keep extra headroom free on the primary device.

2. Communication errors with distributed backends (NCCL P2P)

When vLLM starts, it attempts to initialize direct memory access between both cards via PCIe Peer-to-Peer (P2P). If the motherboard or BIOS settings do not fully support this, it results in initialization errors. This can be resolved by explicitly routing the communication via environment variables:

# Schakel PCIe P2P communicatie expliciet in of pas fallback toe
export NCCL_P2P_DISABLE=0
export NCCL_IB_DISABLE=1
# Indien het moederbord hardwarematige P2P-blokkades opwerpt:
# export NCCL_P2P_DISABLE=1

3. Asynchronous thermal throttling

When the primary card becomes hotter than the secondary card due to restricted airflow, the chip's internal protection throttles down the clock speed. In a sequential pipeline setup, the second card must continuously wait for delayed intermediate results from the first, causing the overall token speed to drop abruptly. Continuously monitoring temperatures and clock frequencies provides insight here:

# Monitor temperaturen, kloksnelheden en actueel vermogen per seconde
watch -n 1 nvidia-smi --format=csv,noheader --query-gpu=index,temperature.gpu,utilization.gpu,power.draw,clocks.current.graphics

Conclusion: Trade-offs and prerequisites for a dual-GPU setup

Combining two GPUs is an effective method to break through the memory limit of a single graphics card and run advanced 70B models locally at workable speeds. By utilizing Pipeline Parallelism within software such as llama.cpp, this approach is also achievable on standard motherboards without specialized NVLink bridges.

The prerequisites—including a robust power supply, adequate cooling, and well-considered memory distribution—require careful planning. Those who have these factors under control achieve a powerful local AI system that combines scalability with full data privacy.