llama.cpp Server: A Lightweight Way to Serve Models

Start llama-server -m model.gguf with no other options and, as of October 2026, it listens on 127.0.0.1:9931, reads the context size from the model file, decides on its own how many layers fit on the GPU, and opens four request slots that share one memory pool. That is the llama.cpp server: a single C/C++ binary from the MIT-licensed llama.cpp project that loads a GGUF model and answers HTTP requests. Most of the work of running it well comes down to understanding what those defaults chose for you, and overriding three or four of them on purpose.

What the bundled server actually gives you

The server ships in the same repository as the inference library, so it tracks new model architectures as soon as the library supports them. Its README lists, among other things:

• OpenAI-compatible routes for chat completions, responses and embeddings
• Anthropic Messages API compatible chat completions
• parallel decoding for several users, with continuous batching on by default
• schema-constrained JSON output and function calling
• speculative decoding with a smaller draft model
• a built-in web chat page
• monitoring endpoints: /health, /slots and an optional Prometheus /metrics

The OpenAI-compatible routes are the reason most people use it. An application already written against a cloud API can usually be pointed at the local server by changing its base URL, a pattern covered in detail in swapping a cloud model for a local one.

Installation is short. Prebuilt packages exist for Homebrew, Winget, Nix, MacPorts and conda-forge, and the project publishes container images, including a CUDA build tagged server-cuda. Building from source is the route when you need a specific GPU backend the packages do not include.

There is also a native /completion endpoint with more sampling controls than the OpenAI format exposes.

The flags that decide memory and speed

Every flag also has an environment variable form, such as LLAMA_ARG_CTX_SIZE for -c, which is handy in a container or a systemd unit. The ones below are the ones worth setting by hand.

Context size: -c / --ctx-size

The default is 0, meaning "use the model's trained context length". For a modern model that can be 32,768 or 131,072 tokens, and the KV cache that holds the context is sized for it up front.

The cost of that cache depends on the model's layer and head counts, and how much VRAM a model needs works the formula through for a 7B model, where 32,768 tokens of 16-bit cache come to about 1.88 GB on top of the weights. What matters for the server is that the cost is linear in -c. If nobody pastes documents longer than 8,000 tokens, -c 8192 hands back about three quarters of it. The -ctk and -ctv flags set the K and V cache data types (default f16, with q8_0 among the allowed values) and shrink it further; test a smaller type on your own prompts before relying on it.

Pick -c from the longest real input, not from the model card. A contract review tool that sends a 20-page document plus instructions needs a different number from an internal chat assistant, and running both from one server means sizing for the larger case.

GPU layers: -ngl / --n-gpu-layers

This sets how many of the model's layers live in VRAM. It accepts a number, auto (the current default) or all. Layers that do not fit run on the CPU from system RAM, which works but is far slower, because generation speed is limited by how fast weights can be read.

Current builds also enable --fit by default, which adjusts any options you left unset so the model fits in device memory, keeping a margin of 1,024 MiB per device and never shrinking context below 4,096 tokens. That is convenient on a desktop. On a dedicated server it can be surprising: a model that silently loaded half its layers on the CPU, or a context quietly reduced, looks like a slow GPU rather than a configuration problem. Read the startup log, which reports how many layers were offloaded and the context per slot, and pin -ngl all and an explicit -c once you know the model fits. If it does not fit, you get an error at startup instead of a slow service.

Parallel slots: -np / --parallel

A slot is one conversation the server can work on at the same moment. Requests beyond the number of slots wait in a queue. The default, -1, means auto, and the server source shows what auto does: it sets four slots and turns on the unified KV cache (-kvu), so all four draw from a single shared pool of context.

The difference between unified and split caches is where people get caught:

Setup	Context per conversation	Behavior under load
-c 32768, -np 4, unified (the auto default)	Any one request can use up to the full 32,768 tokens	Four long conversations at once compete for the same pool
-c 32768, -np 4, --no-kv-unified	Each slot gets 32,768 / 4 = 8,192 tokens	Predictable, but no single request can exceed 8,192
-c 32768, -np 1	32,768 tokens	One request at a time, everyone else queues

The split behavior comes from the library dividing total context by the number of sequences. It catches people who raise -np to serve more staff, keep -c unchanged, and find that long documents now get cut off. If you split the cache, multiply: eight slots that each need 8,192 tokens means -c 65536, and the KV cache memory grows to match. The newer --kv-unified-per-slot N flag caps each slot inside a unified pool and, when -c is not set, sizes the pool to slots times N.

More slots raise total throughput, because continuous batching lets the GPU work on several sequences per step, but each user's tokens per second drops as the batch grows. How many slots a given card can carry is a measurement question, worked through in how many users one GPU can serve.

Network, keys and monitoring

The default bind address of 127.0.0.1 is the safe one: only processes on the same machine can reach it. Changing it to --host 0.0.0.0 exposes the model to the whole network. The server supports --api-key (several keys comma-separated) and --api-key-file (one key per line), checked as a bearer token, and TLS if built with OpenSSL support. A shared key is still a thin control: it cannot tell users apart or log who asked what. Most office deployments keep the server on localhost and put a reverse proxy with real authentication in front. Note that /health is public by design, so a load balancer can check it without a key.

For monitoring, --metrics turns on a Prometheus endpoint with counters such as prompt and generated tokens, average tokens per second, requests processing and requests deferred. A requests_deferred gauge that sits above zero during working hours is the clearest sign that you need more slots or more hardware. The /slots endpoint (on by default) shows what each slot is doing, which is useful for debugging but reveals request state, so consider --no-slots on anything reachable from the network. The built-in web page can be turned off with --no-webui when only applications should talk to the server.

A pinned configuration in one unit file

Put together, the advice above fits in the ExecStart line of a systemd unit. An illustrative setup for four staff on one GPU, each allowed up to 8,192 tokens:

[Service]
User=llama
ExecStart=/usr/local/bin/llama-server \
  -m /srv/models/assistant.gguf --alias assistant \
  -c 32768 -np 4 --no-kv-unified -ngl all \
  --host 127.0.0.1 --port 9931 \
  --metrics --no-slots --no-webui
Restart=on-failure

Every value here is explicit, so an upgrade that changes a default does not change this server. The --alias gives clients a stable model name, which means swapping the GGUF file later does not require touching application code.

Why some teams skip the wrapper

Tools such as Ollama make local models easy: one command pulls a model, and the runtime loads and unloads models as requests arrive. That convenience is a layer of decisions made for you, and some teams prefer to make those decisions themselves.

Reasons to run the llama.cpp server directly:

• Every knob is exposed. Cache types, slot counts, batch sizes, speculative decoding and sampling defaults are all command-line flags, not settings a wrapper may or may not pass through.
• New model support lands here first. Wrappers have to pick up library changes and ship a release; the server is built from the library itself.
• Fewer moving parts. One binary, one GGUF file, one systemd unit. There is no model registry or background downloader, which also makes it easier to run on an air-gapped machine where model files arrive on a drive.
• Predictable memory. With -c, -ngl and -np pinned, the server allocates the same memory on every start.

Reasons not to:

• Model management is yours: downloading, checking and naming files, and editing a unit file to swap models (router mode helps but is newer).
• The flag set changes between releases, so pin the build you tested and re-read the README before upgrading.
• For many concurrent users on datacenter GPUs, a batching-focused engine like vLLM usually gets more out of the hardware. The trade-off is laid out in Ollama vs vLLM.

A reasonable decision rule: if one machine serves a handful of people from GGUF models and you want to know exactly what it is doing, run llama-server directly with pinned flags behind a proxy. If nobody on the team wants to own a unit file, a wrapper is the better choice, and if a queue builds up under real load, that is the point to look at a batching server.

Frequently asked questions

Is the llama.cpp server free to use commercially?

The llama.cpp code, including the server, is released under the MIT license, which permits commercial use. The model you load carries its own license, and that is usually the one with conditions worth reading.

Does llama-server need a GPU?

No. It runs on CPU only, and also supports Apple Metal, CUDA, Vulkan and other backends depending on how it was built. CPU-only serving is practical for small models and low request volume, but generation slows sharply as models get larger.

Why does a command from an old tutorial start on a different port?

Defaults change between releases. Older tutorials commonly show port 8080, while the current README documents 9931 as the default, and slot and offload behavior have moved too. Set --port explicitly in any service file so an upgrade does not break clients.

Can one llama-server process serve several models?

Yes, in router mode, started with --models-dir pointing at a folder of models. Clients choose a model by name in the request, and the server loads it on demand, keeping up to --models-max loaded at once. A single fixed model with -m remains the simpler and more predictable setup.