How to Choose a GPU for a Local LLM

A 32-billion-parameter model in the common Q4_K_M format, which works out to about 4.9 bits per weight, needs roughly 32 x 4.89 / 8 = 19.6 GB just for its weights. Add a cache for a long conversation and the runtime's own working memory, and it fills a 24 GB card. A 16 GB card cannot hold it at all, however fast that card is. That single fact explains most of how to pick the best GPU for a local LLM: memory capacity decides what you can run, memory bandwidth decides how fast it answers, and power and physical size decide whether the card can live in your building. Everything else, including the compute figures on the box, comes after.

The method below works in that order. Cards named as reference points carry specs from the vendors' pages in October 2026; the method outlasts them.

Step 1: Size the memory before anything else

Start with the model you actually plan to run and work out its memory need: weights, plus a KV cache that grows with context length and with the number of simultaneous conversations, plus an allowance for the runtime. The formula and three fully worked examples are in the guide to how much VRAM an LLM needs. For buying, what matters is turning that number into a card class.

Model size	Weights at Q4_K_M	Total with a long context	Card memory that holds it
8B	about 4.9 GB	7 to 10 GB	8 GB only with a short context, 12 to 16 GB is comfortable
32B	about 19.6 GB	22 to 26 GB	24 GB is tight, 32 GB is comfortable
70B	about 42.8 GB	48 to 55 GB	more than any 48 GB card, so 96 GB or two cards

Weights use the 4.89 bits per weight that llama.cpp documents for Q4_K_M. The totals are a range because the cache depends on how much text people feed in: a model that fits a short chat may run out of memory on a 40-page contract.

Two buying rules follow from this step:

• Buy for the model one size up from the one you test with today. Small models are cheap to try and their limits show quickly, and the card should not be what stops the next experiment.
• Treat memory you cannot add later as the expensive mistake. A slow card is an annoyance, but a card that cannot load the model you need is simply the wrong card.

Memory type is a secondary filter. The workstation and datacenter cards named below list error-correcting (ECC) memory, which matters more for a server running unattended for months than for a desk machine.

Step 2: Compare memory bandwidth, not compute

Once a model fits, the speed at which it writes an answer depends mostly on memory bandwidth. To generate each new token, the GPU has to read essentially all of the model's weights from memory. So a simple ceiling for one user's generation speed is:

max tokens per second ~ memory bandwidth (GB/s) / weights read per token (GB)

Take that 32B model, with about 20 GB of weights read for every token, and two current cards that both carry 32 GB, using the vendors' bandwidth figures:

• GeForce RTX 5090, 1,792 GB/s: 1,792 / 20 = about 90 tokens per second, at most.
• Radeon AI PRO R9700, 640 GB/s: 640 / 20 = 32 tokens per second, at most.

Real numbers land below these ceilings, sometimes well below. The ratio between cards is the useful part. Here it is 1,792 / 640 = 2.8 to 1, while the R9700's launch price of $1,299 was about two thirds of the 5090's $1,999. For one person reading answers as they appear, 32 tokens per second is already faster than reading speed, so the cheaper card may be enough. For a queue of colleagues or an overnight batch job, the faster card can finish the same work in roughly a third of the time.

Two limits on this shortcut. Reading a long prompt before the first word appears leans on raw compute rather than bandwidth, and serving several people at once multiplies the demand on both. Published tokens-per-second benchmarks rarely match your model, runtime and prompt length, which is why a ceiling you can calculate from a spec sheet makes a better first filter. If many staff will share the hardware, compare two cards against one bigger card before deciding.

Step 3: Check power, cooling and physical fit

A card that fits the model and runs fast enough still has to work in the machine and room you have.

Power draw

Board power ranges widely. On NVIDIA's comparison page, the GeForce RTX 5060 Ti is rated at 180 W and the RTX 5090 at 575 W, and NVIDIA lists a 1,000 W system power supply requirement for the 5090. For buying, the useful figure is memory per watt: the RTX PRO 6000 Blackwell Max-Q holds 96 GB at 300 W, three times the memory of the 5090 at about half its power. Turning watts into a circuit, a heat load and a power bill is covered in power and cooling for an AI server.

Cooling style

This is where card classes differ most:

• Open-air fans (most consumer cards) push hot air into the case. Fine for one card in a roomy tower, poor for cards side by side.
• Blower coolers push air out of the back of the case, which is why multi-card workstations favor them. Check the cooler type on the spec sheet rather than assuming it from the product class.
• Passive cooling means no fan at all. The NVIDIA L40S, for example, is listed as passively cooled. It depends on a server chassis with strong front-to-back airflow and will overheat in an ordinary desktop tower.

Size and slots

A dual-slot card leaves room for a second one later; a three-slot consumer flagship usually does not. Partner versions of consumer cards are often thicker and longer than the reference design, so check the exact model against the case.

The three card classes

Most buyers choose a class first and a model second. Examples below use specs from the vendor pages in the sources.

	Consumer	Workstation	Datacenter
Example	GeForce RTX 5090	RTX PRO 6000 Blackwell	H200 NVL
Memory	32 GB GDDR7	96 GB GDDR7 with ECC	141 GB
Bandwidth	1,792 GB/s	1,792 GB/s	4.8 TB/s
Max power	575 W	600 W (300 W Max-Q edition)	up to 600 W
Cooling	fans, open air	fans, dual-slot	air, needs server airflow
Lives in	a desktop tower	a workstation or tower server	a rack server

Consumer cards

Consumer cards usually give the most bandwidth per dollar and are easy to buy, though the RTX 5090's street price has often run well above its $1,999 launch price. The trade-offs are capacity (its 32 GB is the most in NVIDIA's current GeForce lineup), no advertised ECC memory, coolers designed for one card in a gaming case, and driver terms aimed at desktops. A consumer card is a sound choice for one team running models up to roughly 32B.

Workstation cards

Workstation cards trade price for memory and reliability. The RTX PRO 6000 Blackwell carries 96 GB with error-correcting memory on the same 1,792 GB/s bandwidth as the RTX 5090. Its Max-Q edition keeps that memory and bandwidth at 300 W, and NVIDIA says it enables up to four GPUs in one system, 384 GB in total. On the AMD side, the Radeon AI PRO R9700 from Step 2 pairs 32 GB with a 300 W budget. Workstation cards make sense once you need a 70B-class model on one card, or several cards in one box.

Datacenter cards

Datacenter cards such as the L40S (48 GB, 864 GB/s, 350 W, passive) and the H200 NVL (141 GB, 4.8 TB/s) are built for racks and constant load. They need a matching server chassis, are usually sold through server vendors, and only pay off when many users or large models keep them busy. Few small offices need this class.

Software support is part of the hardware choice

Most local LLM runtimes support NVIDIA cards through CUDA first and best. AMD cards work with several popular runtimes through ROCm or Vulkan, but support for a specific card or feature can lag. Before buying anything else, confirm that your exact runtime lists the card as supported, and on which operating system and driver versions. A cheaper card that forces an unfamiliar stack can cost more in staff time than it saved.

Second-hand cards are another way to get more memory per dollar, particularly previous-generation 24 GB consumer cards, but they bring their own risks. The checks are in the guide to buying used GPUs for AI.

The method as a checklist

1. Size the largest model you expect to run in the next 12 to 18 months, at the context length real documents need.
2. Drop every card that cannot hold it. If none in your budget can, pick a smaller model or plan for two cards.
3. Among the rest, take the cheapest card whose bandwidth ceiling clears your users' needs with room to spare.
4. Confirm power, cooling and size fit the machine, and that your runtime supports the card.

If two candidates are still tied, take the one with more memory.

Frequently asked questions

Is a gaming GPU acceptable for business use?

Yes, for many small deployments. The limits are memory capacity, the lack of advertised ECC memory, and coolers designed for a single card in an open case. Read the card's driver and software license terms if you plan to put it in a shared server, because desktop and datacenter terms differ.

Does more compute (TFLOPS) make answers faster?

Partly. Compute speeds up reading a long prompt and serving many requests in a batch, but for one person waiting on a reply, memory bandwidth usually sets the pace. Compare bandwidth first and look at compute only between cards with similar memory.

Do two smaller cards work as well as one large one?

You can split a model across two cards to get more total memory, and most runtimes support it. It does not double speed, it needs a motherboard with enough PCIe slots and lanes, and it doubles the heat. One larger card is usually simpler if the price gap is small.

How long will a GPU bought now stay useful?

Memory capacity ages better than anything else on the spec sheet. Open models keep getting more capable at the same size, so a card that holds a 32B or 70B model today will likely run better models in that size class later, even as newer cards get faster.