Batch LLM Jobs: Processing a Million Records Overnight

A ten-hour night is 36,000 seconds. To classify a million records in that window, a pipeline has to finish about 28 records every second (1,000,000 / 36,000 = 27.8), and keep doing it until morning without a person watching. That one division shapes every decision in batch LLM processing: how the work is queued, how results are written, what happens when a record fails, and whether the hardware can keep up at all.

Batch work is also where a local GPU earns its price. A chat assistant sits idle most of the day; a batch job keeps the card busy for hours, and since nobody waits on any single answer, the server can pack many requests together.

What a batch job looks like

Typical overnight jobs share a shape: many independent records, the same instruction applied to each, and a short structured answer per record. Examples:

• Tag 400,000 support tickets with a product area and an urgency level.
• Pull invoice number, date, supplier and total out of scanned and OCR'd invoices.
• Write a two-sentence summary for each of 50,000 contract files.

Because records do not depend on each other, the job can be split, retried and restarted at any point. If the machine loses power at 3 a.m., a restart should finish the remaining records and touch nothing that already succeeded.

The pipeline in four parts

A dependable batch pipeline has four pieces, and each one exists to handle a specific kind of failure.

1. A job table as the queue

The simplest queue is a database table the business already runs. Each record gets a row with its input reference, a status (pending, running, done, failed), an attempt counter and a timestamp. Workers claim rows, process them and mark them done.

In PostgreSQL, the claim step uses FOR UPDATE SKIP LOCKED. The PostgreSQL documentation describes exactly this use: skipping locked rows "can be used to avoid lock contention with multiple consumers accessing a queue-like table." Several workers can then pull from the same table without picking up the same record twice:

UPDATE jobs SET status = 'running', claimed_at = now(), attempts = attempts + 1
WHERE id IN (
  SELECT id FROM jobs
  WHERE status = 'pending'
  ORDER BY id
  LIMIT 50
  FOR UPDATE SKIP LOCKED
)
RETURNING id, input_ref;

Claim in chunks of a few dozen rows, not one at a time, so the database round trips stay small next to the model calls. Add a sweeper that resets rows stuck in running for longer than, say, ten minutes back to pending. That handles a worker that crashed mid-chunk.

2. Idempotent writes

Idempotent means writing the same result twice leaves the same state as writing it once. This matters because retries and restarts guarantee that some records will be processed more than once.

The pattern is to key every output by the record's own ID and write with an upsert, so a second write replaces or skips instead of duplicating:

INSERT INTO ticket_tags (ticket_id, area, urgency, model, prompt_version)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (ticket_id) DO UPDATE
SET area = EXCLUDED.area, urgency = EXCLUDED.urgency,
    model = EXCLUDED.model, prompt_version = EXCLUDED.prompt_version;

Store the model name and a prompt version next to every result. When someone changes the prompt next month, the rows from the old prompt can be found and rerun, and nobody has to guess which output came from which setup. Never let the batch job write directly into the system of record, such as the CRM or ERP. Write to a results table, check it, then apply it in a separate step that can be reviewed.

3. Retries with a limit

Failures in a batch run come in two kinds, and they need different handling:

Failure	Example	Handling
Transient	Server restarted, request timed out, out-of-memory on one oversized input	Retry after a short wait, up to 3 attempts
Permanent	Corrupt file, input far past the context limit, output fails validation every time	Mark failed with the reason, move on

The attempt counter in the job table enforces the limit. Without it, one poisoned record can loop all night. Retry with a wait that grows each time (for example 5, 20 and 60 seconds) so a restarting server is not hammered. At the end of the run, the failed rows become a short list for a person to review.

4. Structured output validation

A model asked for JSON will occasionally return prose, a missing field, or a category that does not exist. At a million records, "occasionally" means thousands of rows.

The first defense is constrained decoding, where the server only allows tokens that fit a schema. Both common runtimes support it. Ollama takes a JSON schema in the format parameter, and its docs recommend a low temperature (they suggest 0) for predictable output. vLLM supports a JSON schema, a regex, a fixed list of choices or a grammar; for a tagging job, the choice option restricts the answer to exactly one of the allowed labels.

The second defense is validating every result in the worker anyway, against the same schema plus business rules a schema cannot express: the total equals the sum of line items, the date is not in the future, the urgency is one of three values. A record that fails validation gets retried once, then marked failed. Constrained decoding guarantees the shape of the answer, not that the answer is right, so validation is the step that catches a well-formed wrong value.

Throughput arithmetic before the first run

The opening division gave the target: about 28 records per second. Whether the hardware can hit it depends on tokens, not records.

Take an illustrative ticket-tagging job:

• Input per record: 350 tokens of instructions plus 250 tokens of ticket text = 600 tokens
• Output per record: about 30 tokens of JSON
• Records: 1,000,000

That is 600 million input tokens and 30 million output tokens. To finish in 36,000 seconds, the server must generate 30,000,000 / 36,000 = 833 output tokens per second and read 600,000,000 / 36,000 = 16,667 input tokens per second, at the same time.

A single request on a local model might generate a few dozen tokens per second, so one-at-a-time processing is out by an order of magnitude. Batching is what closes the gap: a serving engine runs many requests through the GPU together, and the combined throughput is far higher than any single stream. How many concurrent requests a card handles before it saturates is the subject of how many users one GPU can serve, and the same measurements apply here.

Two things in this example stand out. First, input dominates: 600 million tokens in versus 30 million out. Second, 350 of the 600 input tokens are the same instructions repeated a million times. vLLM's automatic prefix caching reuses the computed state for a shared prompt prefix, so the repeated part is not processed from scratch each time. Its docs note the limit plainly: caching cuts prompt processing, not the time to generate new tokens. Putting the fixed instructions first and the variable record last is what makes the cache effective.

Measure, then multiply

Vendor throughput figures rarely match a specific prompt, model and card. The honest method:

1. Pick 1,000 real records.
2. Run them through the full pipeline (queue, model, validation, write) at the concurrency planned for the night.
3. Time it from first claim to last write. Say it takes 50 seconds: that is 20 records per second.
4. Divide: 1,000,000 / 20 = 50,000 seconds, about 13.9 hours.

That run would not fit in ten hours. The options, roughly in order of cost: shorten the instructions, trim the input to the fields the task needs, raise concurrency if the card is not yet saturated, use a smaller model if the test set shows it is accurate enough, or let the job run across two nights. Each change goes back through the 1,000-record timing. For a repeatable way to take these measurements, see how to benchmark local LLM speed.

Choosing the serving setup

The worker talks to the model over HTTP, but the engine behind it still decides throughput. Ollama is easy to stand up but is built around convenience for a few users; vLLM is built around batching many concurrent requests, which is exactly the batch workload. The trade-offs are covered in Ollama vs vLLM.

vLLM also has a file mode for jobs that need no queue at all. vllm run-batch reads a JSON Lines file, one request per line, in the same format as OpenAI's batch API (each line carries a custom_id, method, URL and request body), and writes results to an output file. The vLLM docs list chat completions, embeddings and score as supported endpoints. This suits a one-off job of a few thousand records. For a million records spread over a night, the job table is still the better fit, because a file mode gives no per-record retries and no way to see progress halfway through.

A first-run checklist

Before the first overnight run, confirm each of these on a small sample:

• Killing a worker mid-run and restarting it finishes the job with no duplicate rows.
• A deliberately corrupt input ends up failed with a readable reason after the attempt limit.
• Every result row carries the model name and prompt version.
• The 1,000-record timing, multiplied out, finishes at least two hours before the results are needed.
• Someone has looked at 100 random outputs and agrees they are right.

If the last check fails, no amount of throughput tuning helps. Fix the prompt or the model choice first, then time it again.

Frequently asked questions

Should the batch job run during business hours too?

It can, if staff also use the same GPU for chat. In that case, give interactive requests priority, for example by running batch workers at low concurrency during the day and raising it after hours. A batch job at full concurrency will make chat responses noticeably slower.

How large should each claimed chunk be?

Large enough that database calls are a small share of the time, small enough that a crashed worker does not strand much work. A few dozen to a few hundred records per chunk works for most jobs. The stale-row sweeper picks up whatever a crashed worker left behind.

Is temperature 0 enough to make reruns identical?

Usually close, but not guaranteed. Batched GPU computation can change the order of floating-point operations, so the same input can occasionally produce a different answer depending on what else was in the batch. vLLM's docs say it does not guarantee reproducible results by default. Store the result you accepted instead of assuming a rerun reproduces it.

What should happen to records that fail every attempt?

Keep them in the job table with the error and the raw model output, then review them as a group. They often share a cause, such as one document type the prompt never anticipated or inputs longer than the context setting, and one fix clears most of them.