AI Agents On-Premise: What Is Realistic Today

Suppose each step of a task goes right 95% of the time. A three-step task then succeeds 0.95 x 0.95 x 0.95 = 0.857, about 86% of the time. A ten-step task succeeds 0.95^10 = 0.599, about 60% of the time. A twenty-step task: 0.95^20 = 0.358. The 95% is illustrative, not a measured rate, but that compounding is the most useful single idea for anyone planning on-premise AI agents: a model can be good at each step and still be a poor choice for a long chain of them.

What an agent is in practice

An agent is a loop. Your code sends the model a goal plus a list of tools it may use, each described by a name and a schema for its arguments. The model replies either with a final answer or with a request to call a tool, such as search_orders(customer_id="C-1042"). Your code runs that call, appends the result to the conversation, and sends everything back. The model reads the result and decides the next step. Repeat until it answers or a limit stops it.

Anthropic's December 2024 guide Building effective agents describes agents as "typically just LLMs using tools based on environmental feedback in a loop," and draws a line that is worth keeping:

• Workflows are systems where models and tools are "orchestrated through predefined code paths." Your code decides the order: classify, then extract, then file.
• Agents are systems where the model "dynamically direct[s] [its] own processes and tool usage." The model decides the order.

Much useful business automation is better built as a workflow with one or two model calls inside it. A workflow fails in predictable places. An agent fails wherever the model takes a wrong turn. The mechanics of the tool call itself (the schema, the model's reply, returning results) are covered in function calling with local models. This article is about when to hand the model the steering wheel at all.

Tasks that suit a local agent

The tasks that work share three traits: few steps, a small set of read-mostly tools, and a result a person can check quickly.

Task	Tools needed	Typical steps	Why it fits
Answer "where is order 5512?" from internal systems	order lookup, shipment lookup	2 to 3	Read-only, short, easy to verify
Gather facts for a reply draft	CRM lookup, document search	2 to 4	Output is a draft a person sends
Fill a form from a document, asking for missing fields	extraction, field lookup	3 to 5	Each field can be validated in code
Look up a part across manuals and the parts list	search over two sources	2 to 3	Wrong answers are cheap and visible
Prepare a weekly summary from tickets and logs	read-only queries	3 to 6	Runs offline, reviewed before sending

In each, a wrong lookup costs only a weak draft that a person reads before anything happens.

Tasks that do not fit, yet

The poor fits are often the jobs agent demos feature:

• Open-ended research over many sources. Twenty searches, each choosing the next, is exactly the long chain the arithmetic above punishes.
• Multi-system changes. "Reschedule the customer, update the invoice, notify the warehouse" touches three systems with write access, and a mistake in step two leaves the records inconsistent.
• Anything involving money movement or permissions. Paying a bill, issuing a refund, adding a user to a group.
• Unbounded browsing. A browsing agent reads untrusted pages, which makes it the main target of the hidden-instruction attacks explained in how prompt injection works.
• Tasks with no clear finish. "Keep the CRM tidy" gives the loop no stopping point and no way to tell success from drift.

Some of these can still be automated, but as fixed workflows where code decides the order and the model only fills in text or a classification at defined points.

Why local models struggle with long plans

The gap between a short task and a long one is not unique to local models, and published benchmarks show it plainly. The tau-bench paper from June 2024 (arXiv 2406.12045) tested agents on simulated customer-service conversations with real tool APIs and policy rules. Even leading function-calling agents such as GPT-4o succeeded on fewer than 50% of tasks, and its pass^k metric, which asks whether an agent solves the same task on every one of k tries, fell below 25% at k = 8 in the retail domain. Newer models score higher on these benchmarks, but the lesson about consistency holds: an agent that sometimes gets a task right is a different product from one that gets it right every time.

METR's study Measuring AI Ability to Complete Long Software Tasks, first posted in March 2025, measured the length of task (in human working time) that models complete half the time. On software tasks it put Claude 3.7 Sonnet at about 50 minutes and found the frontier horizon doubling roughly every seven months since 2019. Both numbers matter: capability is climbing quickly, yet the yardstick itself is a 50% success rate, which is far from what a business process needs.

Local models typically sit behind the frontier on that curve, for practical reasons:

Smaller models lose the thread

The open-weight models that fit on one or two office GPUs are usually a fraction of the size of frontier cloud models. They follow a single instruction well but are more likely to repeat a tool call they already made, forget a constraint from the start of the conversation, or declare the task done early.

Context fills up fast

Every tool result is appended to the conversation. A document search returning five 400-token passages adds 2,000 tokens per call; ten calls add 20,000 tokens before the model writes anything. A local server is often run with a smaller context setting to save memory, and when the window overflows the earliest content, usually the goal and the rules, is what gets dropped or truncated.

None of this rules out local agents. It argues for short loops, a hard cap on iterations, and a test set of real tasks run several times each, so you measure consistency rather than a lucky single pass.

Human approval steps

The simplest reliable control is to split tools into two classes: those the agent may call freely and those that pause for a person.

• Free: reads, searches, lookups, drafts written to a holding area.
• Approval required: sending anything outside the company, writing to a system of record, deleting, changing access, spending money.

The approval screen should show the exact action and arguments the code is about to execute, for example the recipient, the subject and the full body of an email, not a summary written by the model. A summary can be wrong in the same way the action is wrong.

Approval also needs a budget. If an agent asks for sign-off forty times a day on trivial actions, people start clicking yes without reading, and the control is gone. Keep approvals for consequential actions, and if a task needs more than two or three of them per run, it is probably better built as a workflow.

Limiting what an agent can touch

OWASP lists this problem as LLM06:2025 Excessive Agency and names three root causes: excessive functionality, excessive permissions and excessive autonomy. Each maps to a concrete design rule.

1. Fewer tools. Give the agent only the tools the task needs. A tool that reads documents should not also expose delete and modify, and a narrow create_ticket tool is safer than a general command runner or shell.
2. Narrow credentials. Each tool runs under its own service account with the least access that works: a read-only database user, a mailbox account that can create drafts but not send, a file share mounted read-only.
3. Enforcement outside the model. OWASP calls this complete mediation: the downstream system checks every request against policy instead of trusting the model to behave. If the database user cannot write, no prompt can make it write.
4. Act as the requesting user. Where possible, tools run with the permissions of the person who asked, so the agent can never see a file that person could not open.
5. Hard limits in the loop. A maximum number of iterations, a time limit, and a cap on calls per tool per run. A loop that hits the cap stops and reports, rather than trying again.
6. A record of every call. Log each tool call with its arguments and result so a strange outcome can be traced. What an AI audit log should record lists the fields.

Running the model on your own server helps with one more limit: network egress. The inference server and its tools can be firewalled so they reach only the internal systems on the list, which removes whole classes of data leaks no matter what the model decides.

A practical starting rule: write the task out as numbered steps. If there are more than five, or any step writes to a system of record, build it as a fixed workflow with an approval gate, using a tool such as those described in self-hosted workflow automation with AI. Hand the model the loop only for short, read-mostly tasks, and measure it on at least 30 real examples, each run three times, before anyone relies on it.

Frequently asked questions

What should happen when a tool call fails halfway through a run?

Return the error to the model as the tool result, in plain words, so it can try a different lookup or stop and report. Count failed calls toward the iteration cap, and make any write tool idempotent, for example by passing a request ID, so a retried call cannot create the same ticket twice.

Is an agent cheaper to run locally than in the cloud?

Agents use far more tokens per task than a single question, since each loop resends the growing conversation. That makes volume high, which tends to favor owned hardware, but a slow local model running ten loops per task can also take minutes per answer. Measure tokens and time per completed task, not per request.

How can an agent be stopped from looping forever?

Set a maximum iteration count and a wall-clock timeout in your own code, not in the prompt. Also detect repeated identical tool calls and end the run when the same call appears twice with the same arguments.

Can an agent and a person share the same work queue?

Yes, and it is often the safest setup. The agent prepares items, such as a drafted reply with the facts it gathered, and places them in the queue a person already works from. Nothing leaves the queue without that person acting on it.