Running an LLM Without a GPU: What CPU Inference Can Do

An 8-billion-parameter model in the common Q4_K_M format takes about 8 x 4.89 / 8 = 4.9 GB of memory. An ordinary desktop with dual-channel DDR5-5600 memory moves about 89.6 GB per second. Divide one by the other and the ceiling is roughly 18 tokens per second, which is faster than most people read. Try the same with a 70B model at the same quantization, about 42.8 GB, and the ceiling drops to about 2 tokens per second. That pair of numbers is the honest answer to whether you can run an LLM without a GPU: yes, and the model's size divided by the machine's memory bandwidth tells you whether the result is useful or painful.

Memory bandwidth sets the speed, not core count

To generate each new token, the runtime reads essentially every weight of the model from memory once. On a CPU, the arithmetic for that token is cheap compared with the cost of fetching several gigabytes from RAM. So the cores spend most of their time waiting on memory, and the useful ceiling is:

tokens per second ceiling = memory bandwidth (GB/s) / model weights read per token (GB)

Memory bandwidth comes from the number of populated memory channels, the memory speed, and 8 bytes per channel per transfer:

bandwidth = channels x transfer rate (MT/s) x 8 bytes

Machine	Memory	Bandwidth	8B Q4_K_M ceiling (4.9 GB)	70B Q4_K_M ceiling (42.8 GB)
Desktop	2 channels, DDR5-5600	89.6 GB/s	about 18 tokens/s	about 2.1 tokens/s
Single-socket server	12 channels, DDR5-6400	614 GB/s	about 125 tokens/s	about 14 tokens/s

The server row matches what AMD lists for its EPYC 9005 processors in October 2026: 12 DDR5 channels, up to 6400 MT/s, and 614 GB/s per socket. Intel has said select Xeon 6700P models are planned to support 8000 MT/s RDIMMs from late 2026, with faster MRDIMMs for the 6900P family in early 2027. Either way, a server board with twelve channels has about seven times the bandwidth of a desktop, and that is where nearly all of its advantage for LLM inference comes from.

Real speeds land below these ceilings, often well below, so use the table to rule hardware out.

Why more cores stop helping

A 16-core and a 64-core processor on the same twelve-channel platform have the same memory bandwidth. Once enough cores are working to keep the memory busy, adding more makes the queue for memory longer without making it faster. That is why generation speed on a CPU usually flattens out well before every core is in use, and why a cheaper part with fewer cores can be a sensible choice for a box that mostly generates text.

Empty slots cost real speed

The bandwidth figure assumes every channel has a memory module in it. A server bought with four 64 GB modules on a twelve-channel board has 256 GB of capacity and one third of the bandwidth it could have. Twelve 32 GB modules give more capacity and three times the speed. Check how slots are populated before buying, and after delivery sudo dmidecode -t memory lists which slots hold a module and at what speed it runs.

Two-socket servers add a wrinkle. Each socket has its own memory, and a thread reading memory attached to the other socket pays a penalty. llama.cpp has a --numa option, documented as an attempt at optimizations that help on some NUMA systems. Do not assume a second socket doubles the speed; measure it.

Prompt processing is where CPUs fall behind

Generation is only half of a request. Before the first word appears, the model has to read the whole prompt, a phase called prefill or prompt processing. Here the work is dense matrix math across many tokens at once, so it is limited by compute, and this is where the core count, the vector instructions and a GPU all matter.

The llama.cpp README lists AVX, AVX2, AVX512 and AMX support on x86, and the newer instruction sets mainly help this phase. Even so, a CPU reads prompts far more slowly than a GPU, and the gap shows up as waiting time. Plug in the prompt speed you measure on your own machine:

wait before first token = prompt tokens / prompt processing speed

A 2-page email thread, 1,000 tokens, at 100 tokens/s = 10 seconds
A 30-page document, 15,000 tokens, at 100 tokens/s = 150 seconds

Ten seconds for a short task is tolerable. Two and a half minutes before anything appears is not, for a person sitting at a screen. This, more than generation speed, is what rules out CPU-only inference for document question answering and coding assistants that send whole files. llama.cpp lets you tune the two phases separately: -t sets threads used during generation and -tb sets threads for batch and prompt processing, which defaults to the same value.

What CPU-only inference is good for

The pattern behind every good fit is the same: a small model, short prompts, short answers, or no one waiting.

Classification and extraction with small models

Routing a support ticket to one of eight queues, tagging an invoice with a cost center, pulling a date and an amount from a short form, or deciding whether an email is a complaint: these tasks need a few hundred tokens in and a few tokens out. A 3B to 8B model often handles them well, and on a CPU the answer arrives in a few seconds.

Overnight batch work

When a job runs while the office is empty, a slow machine is fine as long as it finishes by morning. Take 20,000 customer emails to label, each with about 400 tokens of instructions plus text and a 10-token answer. Suppose measurement on the box shows 150 tokens per second for prompt processing and 15 tokens per second for generation:

per email: 400 / 150 + 10 / 15 = 2.67 s + 0.67 s = 3.33 s
total: 20,000 x 3.33 s = 66,700 s = about 18.5 hours

That misses a 12-hour window. Cut the instructions so the input is 200 tokens and the math changes:

per email: 200 / 150 + 10 / 15 = 1.33 s + 0.67 s = 2.0 s
total: 20,000 x 2.0 s = 40,000 s = about 11.1 hours

Prompt length, not output, decided whether the job fit, which is typical of CPU batch work. Queueing, retries and validation for jobs like this are covered in batch LLM processing.

Mixture-of-experts models

Mixture-of-experts models hold many parameters but read only a few per token, and that shape suits a CPU with plenty of cheap RAM. Qwen3-30B-A3B, for example, has 30.5 billion parameters in total and 3.3 billion activated per token, according to its model card. At Q4_K_M it needs about 30.5 x 4.89 / 8 = 18.6 GB of RAM, but each token reads only about 3.3 x 4.89 / 8 = 2.0 GB. On the 89.6 GB/s desktop, the generation ceiling is roughly 89.6 / 2.0 = 45 tokens per second. The real figure will be lower, and prompt processing still runs at CPU speed, but a model of this kind gives far better answers than a 3B dense model at a similar generation speed.

Testing, fallback and isolated machines

A CPU box is a cheap place to try prompts before buying a GPU, and a fallback when the GPU server is down.

Where CPU-only inference is unusable

• Large dense models for interactive use. A 70B model at 2 tokens per second on a desktop means a 300-word answer, about 400 tokens, takes over three minutes. Even the twelve-channel server's ceiling of 14 tokens per second is a single-user number.
• Long prompts with someone waiting. Retrieval over internal documents, contract review, summarizing meeting transcripts and coding assistants all send thousands of tokens per request.
• Several people at once. Concurrent users share one pool of bandwidth and one set of cores, and prompt processing for each request queues behind the others.
• Anything with a latency target, such as a chatbot on a customer-facing site.

Quantization helps at the margin, because fewer bits per weight means fewer bytes read per token. The same 8B model at Q8_0, about 8.5 bits per weight, is 8.5 GB and has a desktop ceiling near 10.5 tokens per second instead of 18. The trade-offs between formats are in LLM quantization explained. No quantization level turns a 70B model on a desktop into an interactive tool.

Setting up a CPU box so it performs

• Fill every memory channel, one module per channel, at the fastest speed the platform supports. On many platforms two modules per channel forces a lower memory speed, so check the board manual before adding capacity that way.
• Size the model to bandwidth first, then to capacity. Plenty of RAM lets a large model load, not run quickly.
• Set threads deliberately. Start -t at the number of physical cores and test fewer; give prompt processing its own value with -tb.
• Keep the model in RAM. llama.cpp's load mode can be set to mlock, which forces the system to keep the model in RAM rather than swapping it out.
• Measure both phases on your real prompts before committing to a workload. How to benchmark local LLM speed covers measuring prompt processing and generation separately.

A workable decision rule: compute the generation ceiling from bandwidth and model size, then the prompt wait from your typical prompt length and a measured prompt speed. If no one is waiting on the answer, or the prompt is short and the model is small, a CPU is enough. If a person waits on long prompts, the CPU budget is better spent on the memory a GPU needs to hold your model.

Frequently asked questions

Does adding more RAM make CPU inference faster?

Only if the new modules fill channels that were empty. Most desktop boards have four slots but only two channels, so going from two sticks to four adds capacity, not bandwidth. The channel count is fixed by the processor and board, which is why a jump in speed usually means a different platform rather than an upgrade.

Can a small, cheap GPU speed up a mostly CPU setup?

Often, yes. llama.cpp supports hybrid inference that puts part of a model in VRAM and the rest in system RAM, and even a modest card can take over much of the prompt processing work. The layers left on the CPU still run at system memory speed, so the gain depends on how much of the model fits on the card.

Which runtime should be used for CPU-only inference?

llama.cpp and tools built on it, such as Ollama, are the common choice because they run quantized GGUF models efficiently on ordinary x86 processors and on Apple silicon. Serving engines designed around GPU batching usually treat CPU as a secondary target. Whatever the runtime, the bandwidth ceiling is the same, so the choice changes convenience more than speed.

Is a laptop enough to try a model before buying hardware?

For a 3B to 8B model, yes, as long as the laptop has 16 GB or more of RAM. Laptops usually have dual-channel memory, so the desktop numbers above are a fair guide, though sustained runs may throttle. Treat it as a way to judge answer quality, not throughput.