Multi-GPU Setups for LLMs: When Two Cards Help

A 70-billion-parameter model at the common 4-bit Q4_K_M quantization needs about 70 x 4.89 / 8 = 42.8 GB for its weights alone. No 32 GB card can hold it. Two 32 GB cards give 64 GB, enough for the weights plus roughly 21 GB of room for the conversation cache and runtime overhead. That is the dependable benefit of a multi GPU LLM setup: more memory, so bigger models or longer contexts fit. Whether the second card also makes answers arrive faster depends on how the model is split, how the cards talk to each other, and how many people use the server at once. Often it does not.

What a second card reliably adds

Memory adds up cleanly. Mainstream runtimes spread a model across cards, so two 24 GB cards behave roughly like 48 GB, minus some overhead duplicated on each card.

Speed does not add up cleanly. Generating each new token means reading essentially all of the model's weights from GPU memory, which is why memory bandwidth sets the speed ceiling (the arithmetic is in the guide to how much VRAM an LLM needs). Split a model across two cards and each card holds half the weights. Whether those two halves are read one after the other or at the same time is the whole question, and it comes down to the split method.

Splitting one model: layers versus tensors

There are two basic ways to cut a model across GPUs. Runtimes use different names for them, but the mechanics are the same.

Layer split (pipeline)

The model is a stack of layers. A layer split puts the first half of the stack on card A and the second half on card B. A token's computation runs through card A, hands a small intermediate result to card B, and finishes there.

For one user, the cards take turns. Card A works while card B waits, then the reverse. Take the 70B model on two cards that each have 1,792 GB/s of memory bandwidth, the figure NVIDIA lists for the GeForce RTX 5090:

• Card A reads its 21.4 GB of weights: 21.4 / 1,792 = about 11.9 ms
• Card B reads its 21.4 GB: another 11.9 ms
• Total per token: about 23.9 ms, a ceiling of roughly 1,000 / 23.9 = 42 tokens per second

One card with the same bandwidth and enough memory would read all 42.8 GB in the same 23.9 ms. So for one person typing, a layer split gives the capacity of two cards at the speed of one. The upside is that little data crosses between the cards, and cards of different sizes can share the load unevenly.

This is the default in llama.cpp. Its server documentation describes --split-mode layer as the default that will "split layers and KV across GPUs (pipelined)", and --tensor-split sets the proportion each card gets, for example 3,1 when one card has three times the memory of the other. Ollama follows a related rule: per its FAQ, if a model fits entirely on one GPU it loads there, and only if it does not fit is it "spread across all the available GPUs."

Tensor split (tensor parallelism)

A tensor split cuts every layer in half, so both cards work on every token at the same time. In theory each card reads only 21.4 GB per token, in parallel, halving the time: 11.9 ms, or a ceiling near 84 tokens per second for that 70B model.

The catch is communication. After each layer the cards must combine partial results before moving on, which means many small exchanges per token. That traffic goes over NVLink on datacenter hardware, or over PCIe in most office machines. PCIe 5.0 runs at 32 GT/s per lane, so an x16 link moves roughly 63 GB/s each way (the 128 GB/s headline counts both directions), against 1,792 GB/s inside the card. The exchanges are small, but they happen constantly, and each one adds latency. Over PCIe the real result can land anywhere from a small slowdown to close to double, and only a test on the actual hardware shows where.

vLLM, the common engine for batched multi-user serving, uses tensor_parallel_size for this. Its own parallelism documentation recommends tensor parallelism when a model is too large for one GPU but fits on one machine, then adds a qualifier: for GPUs without NVLink, it suggests pipeline parallelism instead, citing higher throughput and lower communication overhead. llama.cpp offers row and an experimental tensor split mode for the same idea.

Where layer splits do get faster: concurrency

With several requests in flight, a layer split stops idling. While card B finishes one request's token, card A starts on the next request. Under steady concurrent load both cards stay busy, and total throughput across all users can rise well above a single card's, even though no individual answer gets faster. Whether a given office generates that much concurrent load is a capacity question, covered in how many users one GPU can serve.

The other option: two cards, two models

Splitting is not the only use for two cards. If each model fits on one card, running them separately avoids inter-card traffic entirely:

• Two different models. A general chat model on card A, and an embedding model plus a small classifier on card B. Document indexing jobs no longer compete with staff chat for memory.
• Two copies of the same model. One instance per card behind a simple load balancer. Each user gets single-card speed, and the server handles about twice as many simultaneous conversations.

Pinning each instance to a card is ordinary configuration. For Ollama on NVIDIA hardware, the GPU documentation says to set CUDA_VISIBLE_DEVICES to a comma-separated list of GPUs, and recommends UUIDs over numeric IDs because "ordering may vary."

For many small offices this is the better use of a second card: two independent services are easier to reason about than one model stretched across a bus.

The motherboard, power and heat

PCIe lanes and slots

Two x16-length slots on a motherboard do not mean two full x16 connections. Mainstream desktop processors have only enough lanes for one full-width graphics slot plus storage. Installing a second card usually splits those lanes. As one concrete example, the ASUS ProArt X870E-Creator WiFi specification states that with a Ryzen 9000 processor, "When you use both PCIEX16(G5)_1 and PCIEX16(G5)_2, they will run at x8 each", and that enabling a particular M.2 drive slot drops the second card to x4. Its third full-length slot runs at x4 through the chipset.

For a layer split or two independent models, x8 per card is generally fine, because little data crosses the bus once the model is loaded. Loading takes longer, and that is mostly it. For tensor parallelism, link width matters more, and workstation platforms with many more processor lanes become worth their price. Read the board manual's lane-sharing footnotes before buying.

Physical spacing matters too. Large consumer cards are thick, so two may not fit side by side, or the top card breathes the bottom card's exhaust.

Power and heat double

NVIDIA lists the RTX 5090 at 575 W total graphics power, so two cards put 1,150 W of GPU load on the supply before the processor, drives and fans. That alone can outgrow a shared 15 A, 120 V office circuit; the circuit and cooling arithmetic is in the guide to power and cooling for an AI server.

When one bigger card is the better buy

Compare the alternative directly. NVIDIA's RTX PRO 6000 workstation card carries 96 GB of ECC memory with the same listed 1,792 GB/s bandwidth as the 5090, at a 600 W maximum. The Max-Q edition keeps the 96 GB and the bandwidth at 300 W.

For the 70B example, that single card holds the 42.8 GB of weights with about 53 GB to spare for long contexts and parallel conversations. Its single-user ceiling is the same 1,792 / 42.8 = 42 tokens per second as the two-card layer split, with no split to configure, no lane sharing, and half or a quarter of the GPU power draw.

A simple way to compare prices on the day you buy:

cost per GB of VRAM = card price / card memory in GB
two-card option = (2 x card price + extra board, PSU and cooling cost) / total GB

The extras include a board with two usable x8 slots, a bigger power supply, better airflow and maybe a new circuit. If the single large card costs about the same per gigabyte once those are counted, it is usually the better buy.

A decision rule that holds up:

1. If the model you need fits on one card you can afford, buy one card.
2. If you need more memory than that and the workload is one or two users at a time, a single larger card beats two smaller ones on simplicity, power and noise.
3. If you have steady concurrent load from many users, two cards running two copies of a model usually give the most throughput per dollar.
4. Split one model across two cards when no single affordable card holds it, and accept that the gain is capacity, not single-user speed.

The card-class comparison itself, consumer against workstation against datacenter, is covered in the guide to choosing a GPU for a local LLM.

Frequently asked questions

Can two different GPU models work together on one LLM?

Usually yes with a layer split, since llama.cpp lets you set uneven proportions with --tensor-split and vLLM's pipeline parallelism supports uneven splits. The slower card sets the pace for its share of the layers.

Does a second GPU help with long documents?

It can, because the KV cache that holds a long document's context needs memory, and a second card adds memory. The model still has to support that context length, and the runtime's context setting must be raised. Speed for processing the long prompt will not double just because a card was added.

Can two consumer cards be linked with NVLink?

Not the current flagship. NVIDIA's RTX 5090 specification lists NVLink as not supported, so two of them communicate over PCIe, which is why the tensor-split caveats above apply in full. NVLink is mainly a datacenter feature today. Check the spec sheet of any card you are considering rather than assuming support from the price tier.

How can a team tell whether a split is actually helping?

Run the same fixed prompts on one card and on two, measuring time to first token and tokens per second, both with a single user and with several concurrent requests. If two cards only help under concurrency, the setup is behaving like a layer split, and two independent model copies may serve the office better.