Getting Reliable JSON Out of a Local LLM

Ask a local model to "reply in JSON" across 10,000 records, and even an illustrative 2% failure rate leaves 10,000 x 0.02 = 200 records your parser rejects. The failures repeat in familiar shapes: a markdown code fence around the object, a friendly sentence before it, single quotes instead of double, a trailing comma, or an object cut off halfway because the output hit its token limit. Getting LLM structured output as JSON stops being a prompting problem once the runtime enforces it. llama.cpp, Ollama and vLLM can all restrict generation to a JSON schema, which removes the syntax failures entirely. What remains is the harder part: making sure the values inside that valid JSON are right.

How a local endpoint mimics the OpenAI request format in general is covered in OpenAI-compatible APIs for local models. This article stays on the structured output itself.

How constrained decoding works

A language model produces text one token at a time. At each step it scores every token in its vocabulary, and the sampler picks one. Constrained decoding adds a filter before the pick: the runtime compiles your schema into a grammar, tracks where the output currently sits in that grammar, and sets the probability of every token that would break it to zero.

If the output so far is {"status": ", and the schema says status is one of open, closed or pending, the only allowed next tokens are ones that start those three words. The model cannot write "Open " or "in progress", because those tokens are masked out.

That gives two guarantees and leaves two gaps:

• Guaranteed: the output parses, and it has the required keys with the declared types, for the schema features the runtime supports (more on that below).
• Guaranteed: enum fields contain one of the listed values.
• Not guaranteed: the values are true. A required total field will always hold a number; it will not always hold the right number.
• Not guaranteed: the output finishes. If generation stops at max_tokens, the object is still cut off mid-way. Check the response's finish reason, and treat length as a separate failure from a bad answer.

The parameters in each runtime

The names below come from each project's documentation as of October 2026. They have changed before, so check the docs for the version you run.

Runtime	Where the schema goes	Notes
llama.cpp server	json_schema or grammar on /completion; response_format on /v1/chat/completions; -j / --json-schema or -jf / --json-schema-file at startup	Converts a subset of JSON Schema to its GBNF grammar format
Ollama	format in the request body, either "json" or a schema object	Also accepts response_format through its OpenAI-compatible endpoint
vLLM	structured_outputs with keys json, regex, choice, grammar or structural_tag, passed via extra_body; response_format with json_schema	Backend chosen with --structured-outputs-config.backend, default auto

llama.cpp

llama.cpp's own grammar language, GBNF, can describe any text format, and JSON Schema is translated into it. The grammars README states plainly that the schema "is only used to constrain the model output and is not injected into the prompt." The model never sees the schema unless you put it in the prompt yourself, so a model that does not know what supplier_tax_id means will fill it with something shaped like a string and nothing more.

Ollama

Ollama's structured outputs page recommends two things worth copying on any runtime: pass the schema as text in the prompt as well, and lower the temperature, for example to 0. Its examples build the schema from a Pydantic or Zod model, which means the same class can validate the response afterwards.

vLLM

Code written for older vLLM releases may use guided_json, guided_choice and related fields. The vLLM documentation says these were removed in v0.12.0 and replaced by the structured_outputs parameter, so an upgrade can break a working integration with no change on your side. The choice key is handy for pure classification: give it a list of labels and the output is exactly one of them, no JSON wrapper needed.

Schema support is a subset

Every runtime accepts JSON Schema, and none accepts all of it. llama.cpp's README lists the gaps for its converter, including:

• minimum and maximum only work on integers, not on number fields.
• Nested $ref is broken, and patternProperties is not supported.
• A pattern must start with ^ and end with $.
• uniqueItems, not and if/then/else are unlikely to be implemented.
• additionalProperties defaults to false, the opposite of the JSON Schema default.

The line that matters most: "Unsupported features are skipped silently." A schema that says "maximum": 100 on a number field is accepted, the limit is ignored, and a value of 250 comes back without complaint. Two practical rules follow. Keep schemas to plain types, enums, required lists and simple arrays. And enforce every range, format and cross-field rule again in your own validation code, never in the schema alone.

Designing schemas models fill well

Constrained decoding controls shape. The model's chance of putting the right value in each slot depends on how the schema is designed.

Name fields for what they hold. invoice_date beats d1. The field name is part of the text the model generates, and a descriptive name acts as a small instruction at exactly the moment it is needed.

Allow null. A required string field with no null option forces an answer. If a support email never mentions an order, a required order_id still gets filled, with a number from a signature line or a made-up one, because the grammar gives the model no other way out. Declare optional facts as nullable, like order_id in the code below, and treat null as "not found."

Give categories an escape value. An enum of support, sales and billing forces a job application into one of the three. Add other or unclear and route those to a person.

Copy, then convert. Dates and amounts printed in many formats are better copied as strings and parsed in code, a pattern worked through in invoice data extraction with a local LLM.

Put reasoning before the answer. Fields are generated in order. A reason or evidence field placed before label lets the model write out the basis for its choice first, and the label is then conditioned on that text. Placed after, it becomes a justification written for an answer already chosen.

Bound every array. A list with no maxItems can repeat until the token limit. If a document has at most 50 line items, say so. llama.cpp also notes that grammars with many optional repetitions can make sampling very slow.

Keep it flat and small. Each extra level of nesting and each extra field is one more place to be wrong. Splitting a twenty-field extraction into two calls of ten is worth testing when accuracy on the later fields drops.

There is a measurable cost to format restrictions on reasoning-heavy tasks. A 2024 study, Let Me Speak Freely?, found that format restrictions cut reasoning performance and that stricter constraints tended to cost more. When a task needs the model to think, such as weighing several clauses against a policy, a two-step pattern helps: let the model answer in free text first, then run a short constrained call that extracts the result into JSON.

Validate, then retry with the error

Valid syntax is the first of three checks, and each catches a different kind of failure.

1. Parse. With constrained decoding this should never fail. If it does, the cause is almost always truncation; check the finish reason and raise the token limit instead of retrying the same request.
2. Schema validation in code. Run the result through Pydantic, a JSON Schema validator or equivalent, with the full rules the runtime may have skipped: ranges, formats, string lengths.
3. Business rules. Dates in a plausible range, totals that add up, an ID that exists in your system.

When a check fails, retry once or twice with the specific error added to the prompt, such as "due_date 2025-02-30 is not a valid date." A generic "try again" at temperature 0 tends to reproduce the same output. After the retry limit, send the record to a review queue rather than looping.

A minimal sketch with Pydantic against Ollama's native API:

from typing import Literal, Optional
from pydantic import BaseModel, Field, ValidationError
import json
import requests

class Ticket(BaseModel):
    reason: str = Field(max_length=300)
    category: Literal["support", "sales", "billing", "other"]
    order_id: Optional[str] = None

def classify(text, model="your-model", tries=3):
    schema = Ticket.model_json_schema()
    error = ""
    for _ in range(tries):
        prompt = f"Classify this message. JSON schema: {json.dumps(schema)}\n{error}\n\n{text}"
        r = requests.post("http://localhost:11434/api/chat", json={
            "model": model, "stream": False, "format": schema,
            "options": {"temperature": 0},
            "messages": [{"role": "user", "content": prompt}],
        }, timeout=120).json()
        try:
            return Ticket.model_validate_json(r["message"]["content"])
        except ValidationError as e:
            error = f"The previous answer failed validation: {e}"
    return None # goes to the review queue

Here is how the numbers work out, with illustrative rates. Run 10,000 records. Constrained decoding brings parse failures to zero. Say 3% fail schema or business checks: 10,000 x 0.03 = 300 records. If a retry with the error fixes two thirds of them, 200 are recovered and 100 go to a person. The first retry costs 300 extra calls, and a second attempt for the 100 that failed again adds 100 more: 400 calls on top of 10,000, about 4% more GPU time, which is cheap next to 300 bad rows in a database. Measure your own rates on real inputs. Running this over a large overnight batch, with queues and idempotent writes, is the subject of batch LLM processing pipelines.

Before going live, run 200 real inputs and count three numbers separately: parse failures, validation failures and wrong values found by hand review. If wrong values exceed what the review queue can absorb, change the schema or the model before changing the prompt.

Frequently asked questions

Does constrained decoding slow down generation?

It adds work at every token, since the runtime must check which tokens the grammar allows. For simple schemas the overhead is usually small, but deeply nested schemas, large enums and many optional repetitions can slow sampling noticeably. Measure tokens per second with and without the schema on your own hardware.

Can a small 7B or 8B model produce usable structured output?

For extraction from clean text into a small flat schema, often yes, because the grammar handles the syntax and the model only has to pick values. Accuracy drops as schemas grow and documents get longer. Test two model sizes on the same 200 inputs and compare validation failures and hand-checked errors.

Is tool calling the same as structured output?

They use the same mechanism, a JSON schema the model's output must fit, but tool calling adds a choice of which function to call and a different response field. Runtimes configure it separately, and some need model-specific parsers. If you only need data back, plain schema-constrained output is simpler to run and to debug.

Should the schema also go in the prompt?

Yes. The constraint stops invalid tokens but does not explain what a field means, and llama.cpp does not show the schema to the model at all. A short description of each field in the prompt, plus the schema itself, gives the model the context it needs to choose the right value.