Apple Silicon for Local LLMs - Where a Mac Fits

A 70-billion-parameter model quantized to about 4.5 bits per weight needs roughly 39 GB just for its weights: 70 billion x 4.5 bits / 8 bits per byte = 39.4 GB. NVIDIA's top GeForce card, the RTX 5090, has 32 GB. A Mac Studio with an M5 Max chip can be configured with 128 GB of unified memory, and the M5 Ultra version goes to 512 GB, according to Apple's spec page in October 2026. That gap in memory is why people shop for a Mac for local LLM work. What the Mac gives up in exchange is speed on long prompts and on several users at once, and those two costs decide whether it suits your office.

Why unified memory is the selling point

On a PC, the model has to fit in the graphics card's own memory (VRAM). System RAM sits on the other side of a PCIe bus, and once part of a model spills there, generation slows sharply. On Apple Silicon, the CPU and GPU share one pool of memory on the same package, so most of the installed RAM is usable for model weights.

Here is what that pool looks like on the current Mac Studio, from Apple's spec page and announcement in October 2026:

Chip	Max unified memory	Memory bandwidth	GPU cores
M5 Max	128 GB	460 GB/s (614 GB/s with the 40-core GPU)	32 or 40
M5 Ultra	512 GB	1.2 TB/s	64 or 80

In October 2026, MacRumors lists starting prices of $2,499 for the M5 Max model and $5,499 for the M5 Ultra model, and reports that the 512 GB configuration, due in late October, is expected to start above $10,000. To get 128 GB of GPU memory on the PC side, you need either several consumer cards in one chassis or workstation and datacenter-class cards, and both routes bring power, cooling and cost problems of their own. Apple lists the whole Mac Studio at 480 W maximum continuous power. NVIDIA lists the RTX 5090 alone at 575 W.

Not every byte is available, though. macOS reserves part of memory for the system and, by default, caps how much the GPU may wire down. The mlx-lm project documents raising that limit with sudo sysctl iogpu.wired_limit_mb=N, where N is larger than the model but below total memory. The setting comes from the mlx-lm documentation, not from an Apple guide, so treat it as something to test, not a guarantee. A sensible plan leaves 8 to 16 GB for the operating system. For the full sizing method, see how much memory a model needs, including the KV cache.

Two speeds, and the Mac is strong at only one

Local inference has two phases, and they stress different parts of the machine.

Generation is limited by memory bandwidth

When a model writes its answer, each new token requires reading every active weight once, so the same bandwidth-over-model-size ceiling used to compare GPUs applies here:

tokens per second ceiling = memory bandwidth / bytes read per token

M5 Ultra, 70B model at 4.5 bits: 1,200 GB/s / 39.4 GB = about 30 tokens/s
M5 Max (614 GB/s), same model: 614 GB/s / 39.4 GB = about 16 tokens/s

Real speeds land below the ceiling, but the method tells you what to expect. Mixture-of-experts models change the picture in the Mac's favor: they hold many parameters in memory but read only the active experts for each token, so a model with a large total size and a small active size can generate quickly while still needing lots of memory to load. That is exactly the shape a big unified memory pool suits.

Prompt processing is limited by compute

Before the first word appears, the model has to read your whole prompt. That phase, often called prefill or prompt processing, is dominated by raw matrix math, and this is where a dedicated NVIDIA card has historically pulled far ahead of Apple GPUs. Apple says the M5 generation adds a Neural Accelerator to every GPU core and claims up to 4.5x the peak GPU AI compute of the M3 Ultra. That is a vendor figure from Apple's own testing, so measure it on your workload before relying on it.

The cost shows up as waiting time. Put your own measured number in for P:

wait before first token = prompt tokens / prompt processing speed (P)

A 40-page contract at about 500 tokens per page = 20,000 tokens
If P = 500 tokens/s: 20,000 / 500 = 40 seconds
If P = 2,000 tokens/s: 20,000 / 2,000 = 10 seconds

A short chat question barely notices this. Document Q&A, retrieval with long context, coding assistants that send whole files, and summarizing transcripts all live in the long-prompt world. If your use case is "paste in a long document and ask about it", prompt processing speed is the number that decides whether staff find the tool usable. The method for measuring both speeds separately is in how to benchmark a local LLM.

Concurrency is the other cost

A Mac serving one person is a different job from a Mac serving a team. Two things limit how many people it can handle at once.

Memory per conversation. Every active conversation keeps its own KV cache, and that cache grows with context length. Ollama's documentation says it directly: required RAM scales with OLLAMA_NUM_PARALLEL multiplied by OLLAMA_CONTEXT_LENGTH, and OLLAMA_NUM_PARALLEL defaults to 1. The llama.cpp server exposes the same idea through its -np slot count and a shared KV buffer. A model that fits comfortably with one 32,000-token conversation may not fit with four.

Compute shared across requests. Prompt processing for several users competes for the same GPU. Take the contract example above at P = 500 tokens/s. If four people submit 20,000-token prompts at the same moment, the machine has 80,000 tokens to read. The last person in line can wait around 160 seconds for a first word, even though each person alone would wait 40.

The serving software matters too. vLLM, the engine many teams use for batched multi-user serving on NVIDIA hardware, lists macOS support as experimental and CPU-only, and points Apple GPU users to a separate community plugin built on MLX. On a Mac the practical choices are llama.cpp, Ollama and MLX-based servers. They work well for one user or a handful, but they are not built for the dense request batching a busy shared service relies on. The trade-off between those engines is covered in Ollama vs vLLM, and the sizing method for many users is in how many users one GPU can serve.

The operational side

A Mac Studio is quiet, small and draws little power at idle, which makes it easy to put in an office without a server room. It is also a desktop computer running a desktop operating system, and that shows when you run it as a service:

• There is no out-of-band management controller, so a hung machine needs someone to press the button.
• macOS updates, sleep settings and login behavior need to be set so the model server restarts on its own after a reboot. On macOS, Ollama's documentation says environment variables for the app are set with launchctl setenv, which is easy to forget and lose after a reinstall.
• Memory is fixed at purchase. If you buy 96 GB and later want a larger model, the answer is a new machine, not a new module.
• Linux-first tooling (container GPU passthrough, most production serving stacks) does not carry over directly.

None of these rule a Mac out. They mean it behaves like a well-kept workstation, not like a rack server.

When a Mac is the sensible box, and when it is not

A Mac fits well when most of these are true:

• One person or a small team of a few people, with requests that rarely overlap.
• The goal is to run a large model, or a large mixture-of-experts model, that would otherwise need several GPUs.
• Prompts are mostly short to moderate: drafting, chat, classification, short document questions.
• The machine sits in an ordinary office, where noise, heat and a standard power outlet matter.
• Evaluation and prototyping, where loading many different models matters more than serving speed.

Look at a GPU server instead when:

• Ten or more people will use it during the same working hours.
• Most requests carry long documents, retrieval context or whole source files.
• You need high-throughput batch work overnight, where batched serving engines make better use of the hardware.
• You want the standard Linux serving stack and remote management.

A simple decision rule: measure prompt processing speed on a Mac you can borrow or rent, take your typical prompt length, and multiply by the number of people likely to submit at the same moment. If that wait is acceptable, the Mac's memory advantage probably outweighs everything else. If it is not, memory size was never your bottleneck, and a GPU box is the better buy.

Frequently asked questions

Is an M5 Max with 128 GB enough, or is the Ultra worth it?

It depends on the model you want to run and on bandwidth. 128 GB holds a 70B model at 4-bit with room for context, but the Ultra has about twice the bandwidth of the fastest M5 Max (1.2 TB/s against 614 GB/s), so the same model generates roughly twice as fast. Pay for the Ultra if you need models above about 100 GB or faster generation. Headroom you will never fill is not a reason on its own.

Can a MacBook Pro serve as the office AI machine?

It can run the same software, but a laptop closes, travels, sleeps and throttles under sustained load, all of which make it unreliable as a shared service. It is a reasonable way to test models and measure prompt processing speed before buying a dedicated machine.

Does the choice between llama.cpp, Ollama and MLX matter on a Mac?

They share the same hardware limits, so none will turn a Mac into a multi-user server. Speeds differ by model and version, and MLX is Apple's own framework, so test two of them on your real prompts. Ollama is the simplest to set up, while llama.cpp exposes more tuning flags.

Can several Macs be combined to run a bigger model?

Yes, with some runtimes: llama.cpp's server accepts a list of RPC servers to spread a model across machines, and mlx-lm lists distributed inference through mx.distributed. Expect to gain memory more reliably than speed, because the machines have to pass data between them over a link far slower than unified memory. For most small offices, one Mac with enough memory is simpler than two that need to cooperate.