Insights

How Many Users Can One GPU Serve? A Sizing Method

A capacity planning method for a shared local LLM, turning staff headcount into concurrent requests and checking them against tokens per second you measure on your own GPU.

Share:
How Many Users Can One GPU Serve? A Sizing Method

Fifty staff do not mean fifty requests at once. If each of them asks a local model 20 questions in an 8-hour day, that is 1,000 requests spread over 28,800 seconds, or one new request every 29 seconds on average. If an answer takes 15 seconds to finish, the server is working on about half a request at any moment. So the honest answer to how many users one GPU can serve for an LLM is a calculation, not a number: how many requests arrive in your busiest hour, how long each one takes on your hardware, and how many the server can work on at the same time before answers get too slow.

The calculation below uses no published throughput figures, because those depend on the card, model, quantization, prompt length and serving software. The inputs come from measuring your own setup.

Headcount, requests and concurrency

Three numbers get confused in capacity talk:

  • Headcount: people who have access. Useful for licensing, not for sizing.
  • Arrival rate: requests per second during the busiest period. This is what loads the server.
  • Concurrency: requests in progress at the same moment. This is what the GPU has to hold in memory and compute on.

A standard result from queueing theory, Little's Law, links them: average concurrency equals arrival rate times average time per request. In the opening example, 1,000 / 28,800 = 0.035 requests per second, times 15 seconds, gives 0.52 requests in flight on average.

Daily averages flatter the server. Usage bunches up after a morning meeting or at month end, so size for the peak hour. If a third of the day's 1,000 requests land in one hour, the rate is 333 / 3,600 = 0.093 per second, and concurrency is 0.093 x 15 = 1.4. Still small, but nearly three times the daily average.

The other variable is how much each person's question costs. A chat question with a two-line prompt is cheap. A document question that sends 6,000 tokens of retrieved text is not, and an agent that calls the model eight times to answer one question is eight requests, not one.

What a GPU does with several requests at once

Generating a token means reading the model's weights from GPU memory. For one user, the card spends most of its time moving weights, not computing. Serving software exploits this with batching: one pass over the weights produces the next token for several requests at once. Total output across all users rises as the batch grows, while each user's own speed falls somewhat. That trade, aggregate throughput against per-user speed, is the whole sizing problem.

Batching has a ceiling, and it is usually memory. Every active request keeps its own KV cache, which grows with its context length. Once the cache space is full, new requests wait. The arithmetic for that is in how much VRAM a model needs, and the reason long contexts eat capacity is covered in why long documents strain local LLMs.

The common runtimes handle this differently, and their defaults matter:

Runtime Concurrency setting Documented default When it is full
Ollama OLLAMA_NUM_PARALLEL 1 request per model Requests queue, up to OLLAMA_MAX_QUEUE (default 512), then HTTP 503
llama.cpp server -np / --parallel slots -1 (auto); continuous batching on Requests wait for a free slot
vLLM --max-num-seqs Set it explicitly Requests wait; under cache pressure, running requests are preempted and recomputed later

Per the Ollama FAQ, required memory scales with OLLAMA_NUM_PARALLEL x OLLAMA_CONTEXT_LENGTH, so four parallel slots at a 2K context reserve memory for 8K. With the default of 1, a second user simply waits for the first answer to finish. vLLM pre-allocates a fraction of GPU memory for the model and cache, controlled by --gpu-memory-utilization (default 0.92), and its startup log reports how many tokens of KV cache fit and roughly how many full-length requests that allows. That last figure is a useful first ceiling, since short requests use less than the maximum.

Choosing between the two kinds of server is covered in Ollama vs vLLM; the method below works with either.

Measure the curve, do not borrow it

You need one table, measured on the exact model file, quantization and context setting you plan to run, with prompts that look like real traffic:

Concurrent requests Time to first token (p95) Tokens/s per request Total output tokens/s
1
2
4
8
16

Fill it by sending a fixed number of requests while capping how many are in flight. vLLM ships a client for this: vllm bench serve with --max-concurrency caps outstanding requests, --num-prompts sets the run length, and it reports time to first token, time per output token and output throughput, per its benchmark CLI docs. Any load tool that fires parallel streaming requests and records timings also works. The details of a fair run, warm versus cold and fixed prompts, are in how to benchmark a local LLM.

Two things to watch in the results. Total throughput climbs with concurrency, then flattens; past that knee, more parallel requests only make everyone slower. And time to first token rises with load, because new prompts compete with answers already being generated. Pick the highest concurrency where per-request speed and first-token time are still acceptable to your users. Write that limit down, set the runtime to it, and call it C.

A back-of-envelope capacity check

The numbers below are placeholders to show the arithmetic. They describe no particular card or model. Replace them with your measured table.

Step 1: peak arrival rate. A 120-person office expects about 240 requests in its busiest hour: 240 / 3,600 = 0.067 requests per second.

Step 2: request shape. Logs from a pilot show an average answer of 400 output tokens.

Step 3: service time at your chosen limit. Suppose the table says that at 4 concurrent requests, each one gets 25 tokens per second and first token arrives in 1.2 seconds. One answer takes 1.2 + 400 / 25 = 17.2 seconds.

Step 4: expected concurrency. Little's Law: 0.067 x 17.2 = 1.15 requests in flight on average, against a limit of C = 4.

Step 5: throughput utilization. Demand is 0.067 x 400 = 26.7 output tokens per second. At 4 concurrent requests the server delivers 4 x 25 = 100. Utilization is 26.7 / 100 = 27 percent.

Step 6: bursts. Averages hide collisions. If requests arrive independently, the number in flight roughly follows a Poisson distribution with mean 1.15. The chance of more than 4 at once is 1 minus the sum of the probabilities for 0 through 4: 1 - (0.317 + 0.364 + 0.209 + 0.080 + 0.023) = 0.007, under 1 percent of the time. When it happens, the fifth request waits a few seconds for a slot.

That server has room. To find the ceiling, increase the arrival rate until utilization approaches your comfort level. Leave real headroom, because queueing delay grows steeply near full load. In the simplest queueing model, time in the system is the service time divided by (1 - utilization): 2x at 50 percent, 5x at 80 percent, 10x at 90 percent. Batching servers do better than that model, but the shape holds.

Working backward with a 60 percent target, 0.6 x 100 = 60 tokens per second, or 60 / 400 = 0.15 requests per second, or 540 requests in the peak hour. At 2 requests per person in that hour, that is about 270 staff. At that load, concurrency is 0.15 x 17.2 = 2.6, so measure the 8-request row too before trusting the result.

What breaks the estimate

  • Long prompts. Document Q&A sends thousands of input tokens per request. Prompt processing then dominates time to first token, and each request holds more KV cache, which lowers C. Measure with realistic retrieved context, not two-line questions.
  • Agents and tool calls. One question can become several model calls. Count calls, not questions.
  • Batch jobs on the same card. An overnight extraction job that runs into the morning takes slots from interactive users. Separate them by time or by priority.
  • Model swaps. If staff can pick between models and they do not all fit in memory at once, Ollama queues new requests until the requested model can be loaded. Every swap is a pause for everyone.
  • Longer answers over time. Once people use the tool for full drafts, average output length and service time grow. Re-run the arithmetic each quarter from real logs.

Watching capacity in production

The running server tells you whether the estimate holds. vLLM exposes Prometheus metrics including vllm:num_requests_running, vllm:num_requests_waiting, vllm:kv_cache_usage_perc, vllm:num_preemptions and a time-to-first-token histogram, per its metrics docs. llama.cpp's server reports per-slot state at /slots by default and Prometheus metrics at /metrics when started with --metrics. With Ollama, a 503 response means the queue limit was reached.

A practical decision rule: if requests are waiting during most peak hours, or time to first token at p95 exceeds what users tolerate, first check for avoidable load (oversized context setting, batch work at the wrong time, needless model swaps). If the load is real, a second GPU running a second copy of the same model behind a load balancer roughly doubles capacity, which is a cleaner step than squeezing more parallel slots out of one card.

Frequently asked questions

Does a smaller or more heavily quantized model serve more users?

Usually yes. Smaller weights mean less data to read per token and more memory left for KV cache, so both per-request speed and the number of concurrent requests can rise. The cost is answer quality, so test the smaller model on your own tasks before switching for capacity reasons.

Why does the answer slow down when colleagues are also using the tool?

Batching shares each pass over the model's weights among all active requests, so total output goes up while each request's share goes down. New prompts also need processing time that competes with answers already being written. Some slowdown under load is normal; a large one means concurrency is set above the knee of your measured curve.

Will two GPUs in one machine double the number of users?

Only if each card holds a full copy of the model and traffic is spread between them. If the model is split across both cards because it does not fit on one, the extra card mostly buys memory, and capacity rises far less than two-fold.

Is it better to set a high parallel limit and let requests share the GPU?

Not past the point your measurements support. Each slot reserves cache memory, and beyond the knee every user gets slower without the server finishing more work. A moderate limit plus a short queue gives more predictable response times.

Ready to Modernize Your Business?

Get your AI automation roadmap in minutes, not months.

Analyze Your Workflows →