Load Balancing Several LLM Servers

Put three inference servers behind a gateway that probes each one every 30 seconds and marks a server down after three failed probes. When one server's GPU faults, the gateway can keep sending it traffic for up to 90 seconds. At a peak of 2 requests per second, that is 180 requests, and a third of them, 60, land on the broken box. Load balancing LLM servers is mostly about shrinking that number: noticing failure fast, retrying safely, and sending each request to the server that can answer it best.

How many users a single card can carry is a separate question, worked through in how many users one GPU can serve. This article starts at the point where the answer is "more than one box", or where one box going down is no longer acceptable.

What the gateway in front does

Applications call one address, usually an OpenAI-compatible endpoint, and a gateway decides which server handles the request. Its jobs are to know which servers are healthy, which server hosts which model, which eligible server is least loaded or already holds the conversation in cache, and what to do when a request fails.

Three families of software can sit in that spot, and they differ mainly in how much they understand about LLM traffic.

Gateway type	Example	Health checks	Reads the model field	Cache-aware routing
General HTTP proxy	nginx (open source)	Passive only; active checks are in the commercial edition	No, routes by path or host	Only via a hash on a header
LLM-aware proxy	LiteLLM	Background checks that send a real request	Yes	Session affinity by header or key
Engine-specific router	vLLM production-stack router	Depends on deployment	Yes	Session, prefix-aware and KV-aware modes

A general proxy suits two identical servers. An LLM-aware proxy earns its place once several models or real health probing are involved. Engine-specific routers suit sites already running Kubernetes and vLLM.

Health checks that send a real prompt

The cheapest health check asks a server whether its process is up. For an LLM server that answer is close to useless: a vLLM issue asking for a GPU-verifying readiness endpoint describes a GPU fault where /health keeps returning 200 and the load balancer keeps feeding a box that cannot generate.

What each runtime's built-in endpoints can and cannot tell a gateway:

• llama.cpp server: per its README, GET /health returns HTTP 503 while the model is loading and 200 once it is ready. That makes it a good readiness signal after a restart. GET /slots?fail_on_no_slot=1 returns 503 when every slot is busy, which a gateway can use to steer around a saturated server.
• Ollama: GET /api/ps lists the models currently loaded into memory. It shows whether the expected model is resident, not whether it can still generate.
• vLLM: /health reports that the engine process is alive, with the GPU-fault gap described above.

The check that catches the rest sends a short real prompt with a known answer and a tight token limit. How to write that probe is covered in the end-to-end probe for a self-hosted LLM; the difference here is that a failure takes the server out of rotation instead of paging a person.

LiteLLM does this natively. Its /health endpoint runs a real test request against every configured model. With background_health_checks: true the checks run on a loop, every 300 seconds by default (health_check_interval), and enable_health_check_routing: true removes a failing deployment from routing until a later check passes. The default interval matters: at 300 seconds, the opening example's exposure window grows from 90 seconds to as much as five minutes. An interval of 30 to 60 seconds costs a few tokens per server per minute, which is nothing on owned hardware.

The open-source edition of nginx has no active health checks; the health_check module is part of the commercial subscription. It relies on passive detection instead: with the defaults max_fails=1 and fail_timeout=10s, one failed request marks a server unavailable for 10 seconds. That works for a server that refuses connections. It does not catch a server that answers HTTP 200 with garbage or hangs on generation, which is why a separate probe script, or an LLM-aware gateway, belongs next to it.

Routing by model

Requests must reach a server that already has the right model loaded, since loading one on demand takes seconds to minutes.

An LLM-aware gateway routes on the model field in the request body. In LiteLLM, every entry in the model list has a public model_name; several entries with the same name form one group, and the router balances within the group. Two servers both hosting the chat model and one hosting the coding model would look like this:

model_list:
  - model_name: office-chat
    litellm_params:
      model: openai/office-chat
      api_base: http://10.0.0.11:8000/v1
  - model_name: office-chat
    litellm_params:
      model: openai/office-chat
      api_base: http://10.0.0.12:8000/v1
  - model_name: office-code
    litellm_params:
      model: openai/office-code
      api_base: http://10.0.0.13:8000/v1

A plain HTTP proxy cannot read the JSON body cheaply, so the usual workaround is a different path or hostname per model, which pushes the routing decision into every application's configuration. The design question to settle early is redundancy per model. In the layout above, the coding model has one server, so losing that box means the coding assistant is down while chat carries on. That should be a decision, not an accident.

Choosing where each request goes

Round robin fits LLM traffic poorly

Round robin assumes requests cost about the same. LLM requests do not. A yes-or-no classification might produce 5 tokens; a contract summary might read 8,000 and write 1,500. With strict rotation, one server can end up holding three long summaries while its neighbor sits idle between short questions. Balancing on current load fits better: nginx's least_conn sends each request to the server with the fewest active connections, and LiteLLM's least-busy strategy picks the deployment with the fewest ongoing calls. LiteLLM's default, simple-shuffle, is a weighted random pick, which is fine when servers are evenly loaded and loses to least-busy when request sizes vary widely.

Keep conversations where their cache is

Every chat turn resends the whole conversation. Both llama.cpp (cache_prompt, on by default) and vLLM (automatic prefix caching) keep the processed prefix of earlier requests in GPU memory, so a follow-up that shares that prefix only has to process the new part.

The arithmetic shows why this matters for a balancer. Take a conversation with 6,000 tokens of history, mostly a pasted document, and a 200-token follow-up question. On the server that handled the previous turn, about 200 new tokens need processing. On any other server, all 6,200 do, 31 times as much prompt work, and the user feels it as a longer wait for the first word. A balancer that spreads turns of one conversation across servers throws that cache away on most turns.

Three ways to keep requests together:

• Sticky by header or key. nginx's hash $http_x_session_id consistent; maps each session ID to a server, and the consistent parameter means adding or removing a server remaps only a few keys. LiteLLM's session_affinity pre-call check pins a conversation to one deployment using the x-litellm-session-id header. A pin lapses after an hour without traffic by default (deployment_affinity_ttl_seconds: 3600).
• Sticky by user. LiteLLM's deployment_affinity pins by virtual key or user instead. Simpler for applications that cannot send a session ID, coarser in balance.
• Prefix-aware routing. The vLLM production-stack router accepts --routing-logic values including session, prefixaware and kvaware, which route on the prompt itself rather than on a label the application has to supply. This shines when many requests share a long system prompt or the same retrieved documents.

The trade-off is balance. Stickiness can pile several long-running conversations onto one server while others idle. Session pinning with a time limit, plus least-loaded routing for requests without a session, keeps most of the cache benefit.

Failover without surprises

Generation requests are POSTs. Per the nginx proxy documentation, requests with non-idempotent methods such as POST are not passed to the next server once they have been sent to an upstream, unless non_idempotent is added to proxy_next_upstream. For pure generation, retrying is usually safe, since nothing is written. For an agent call that triggers a tool, a retry can run the action twice, so leave that path without automatic retries.

Streaming cannot be retried halfway. Once a server has streamed half an answer to the user, no gateway can quietly switch servers. Failover only covers failures before the first token. Streaming also needs proxy_buffering off in nginx, or tokens arrive in clumps.

Timeouts are idle timers. nginx's proxy_read_timeout defaults to 60 seconds between two reads, not for the whole response. A long document with a slow prefill can produce no bytes for longer than that, and the user gets a 504 while the server is still working. Set it to the longest expected time to first token, plus margin.

Fallbacks must respect the data rule. LiteLLM's fallbacks and order settings move a failed request to another model group or a lower-priority deployment. If the fallback target is a cloud model, an outage quietly sends confidential text off-site. Fallbacks should point only at other local servers unless the request's data class allows otherwise, as covered in splitting work between local and cloud models.

Drain before maintenance. Remove a server from the gateway, let in-flight requests finish, then stop it.

A sensible first layout for two or three servers: one LLM-aware gateway, a real-prompt health check every 30 to 60 seconds, least-busy routing with session affinity for chat, at least two servers for every model staff depend on, and fallbacks that never leave the building.

Frequently asked questions

Does the gateway itself become a single point of failure?

Yes, if it runs as one process on one machine. The usual fix is two gateway instances behind a floating IP address or a DNS name with two records. When gateways run in pairs, check which routing features keep state in memory: LiteLLM's routing documentation, for example, uses Redis to share usage counters between instances.

Can servers with different GPUs sit behind the same gateway?

They can, but equal shares will overload the slower one. Give each server a weight that matches its measured throughput, or use a load-based strategy such as least connections, which shifts work away from the slower box as its queue grows.

Is load balancing worth it with only two servers?

For availability, yes: a model update or a driver problem on one box no longer stops everyone. For capacity, a second identical server adds its own batch of concurrent requests, and the main cost is one more piece of software to configure and watch.