When should you use an AI agent? When the sequence of steps cannot be known in advance, the model can observe the result of each action, mistakes are recoverable, and you can afford the latency and cost of a loop. If any of those is missing, you want a workflow. If the task fits in one call with the right context, you want a single prompt.
That ordering is the whole framework: start with the simplest architecture, and move up only when you can name the requirement that the simpler one fails. Each step up buys flexibility and charges you in latency, tokens, variance, debuggability, and evaluation difficulty. The charges are certain. The benefit is only real if the task needs it.
Three tiers, defined
The terms are used loosely, so here are the definitions we work with. They follow the distinction Anthropic drew in its essay "Building effective agents", which has become common usage: what matters is who controls the path.
Single prompt. One model call. Input, instructions, and any retrieved context go in; the answer comes out. Retrieval-augmented generation is still this tier: your code fetches the context, the model makes one call. So is a call with a structured-output schema.
Workflow. Multiple model calls, orchestrated by your code along a path you defined in advance. The model does the work inside each step; it does not decide what the steps are. Common shapes:
- Chaining. Step one's output feeds step two: extract, then validate, then summarize.
- Routing. A classifier, which may be a model or plain rules, sends the input down one of several fixed branches.
- Parallelization. Split a job into independent pieces, run them concurrently, merge the results. Or run the same task several times and vote.
- Evaluator loop. One call generates, another critiques against stated criteria, and the loop repeats up to a fixed limit.
Agent. The model runs in a loop, choosing which tool to call next based on what it has observed so far, until it decides the task is done or a budget runs out. The path is determined at run time by the model, not at design time by you.
The line between a workflow with a loop and an agent is control. If your code decides what happens next, it is a workflow. If the model does, it is an agent.
What each step up costs
| Single prompt | Workflow | Agent | |
|---|---|---|---|
| Who decides the path | Nobody; there is one step | Your code | The model |
| Model calls per task | 1 | Fixed or bounded | Variable, often many |
| Latency | Lowest | Sum of the critical path | Highest and unpredictable |
| Token cost | Lowest | Predictable multiple | Variable; context grows each turn |
| Run-to-run variance | Low | Moderate, confined to steps | High; different paths on the same input |
| Debugging | Read one prompt and one output | Inspect the step that failed | Read a full trace and reason about choices |
| Evaluation | Input-output test set | Per-step plus end-to-end | Outcomes plus trajectories, in an environment |
| Failure mode | Wrong answer | Wrong answer at a known step | Wrong answer, wrong action, loop, or runaway cost |
Two of these deserve emphasis.
Errors compound. If each step in a chain is right most of the time, the chain as a whole is right less often than any single step. A workflow lets you put validation between steps to stop errors propagating. An agent has to notice and correct its own mistakes, which works only when the environment gives it clear feedback.
Agent cost is not linear in steps. Each turn typically re-sends the accumulated history of tool calls and results, so the context grows as the loop runs. A task that takes twice as many steps can cost considerably more than twice as much.
Six questions that decide it
Ask them in order. The first "no" usually tells you where to stop.
1. Is the path known in advance? If you can draw the flowchart before seeing the input, even a flowchart with branches, it is a workflow or a single prompt. Agents are for tasks where the next step depends on what the last step discovered, in ways you cannot enumerate.
2. How many steps, and how many distinct skills? One transformation with all the context available is a single prompt. Several distinct jobs (classify, then extract, then check, then draft) do better as separate steps, because each prompt can be focused, tested, and assigned an appropriately sized model.
3. Can the model observe and verify results? Agents do well where actions produce unambiguous feedback: tests pass or fail, a query returns rows or an error, a file exists or does not. Where feedback is absent or subjective, the loop has nothing to steer by and tends to declare success.
4. What does a wrong action cost? Reading is cheap to get wrong. Sending an email, moving money, deleting a record, or changing production configuration is not. The higher the cost and the lower the reversibility, the more of the path should be fixed in code, with a human approving the consequential step.
5. What is the latency budget? A user waiting on a response will tolerate a call or two. A multi-step loop belongs in background work, where minutes are acceptable.
6. Can you evaluate it? If you cannot build a test set and score the system, you cannot improve it or safely change it. Single prompts are easy to evaluate, workflows manageable, agents hard. If you are not prepared to build the evaluation harness an agent requires, you are not prepared to run one. Our article on the three measurements we insist on covers what that harness should track.
Worked examples by tier
Single prompt
- Classify an inbound support ticket into one of twelve categories.
- Extract parties, dates, and governing law from a contract into a schema.
- Answer a policy question from the five most relevant retrieved passages.
- Rewrite a technical paragraph for a non-technical reader.
If one of these is performing poorly, the fix is nearly always better context, clearer instructions, examples, or a schema, not more architecture.
Workflow
- Invoice processing: extract fields, validate against the source and the PO system, apply approval rules, draft an exception note when needed. The path is fixed and the decisions belong to rules, as described in rule engines are back.
- Support triage: route by category, then run a category-specific prompt with its own retrieval source and tone guidelines.
- Long-document review: split by section, analyze sections in parallel against a checklist, merge the findings, then run one pass to remove duplicates.
- Report drafting with an evaluator loop: generate, score against a rubric, revise, at most three rounds.
Agent
- A coding task across an unfamiliar repository: the files to read, the change to make, and the tests to run depend on what the agent finds, and the test suite gives clear feedback.
- Investigating a data-quality anomaly: the model queries tables, inspects results, forms a hypothesis, and queries again. The next query cannot be scripted in advance. Access is read-only.
- Open-ended research across many sources, where which source to consult next depends on what the previous one said.
Notice what the agent examples share: an unknown path, observable feedback, and actions that are either sandboxed or read-only.
Signs you have over-built
The agent always takes the same path. Look at fifty traces. If the tool sequence is identical in nearly all of them, you have a workflow that pays agent prices. Hard-code the sequence.
Most of the system prompt is procedure. "First do A, then always do B, never do C before D" is a flowchart written in prose and enforced probabilistically. Move it into code.
You cannot explain a failure without reading a long trace. That is the debugging cost showing up. If it happens weekly, the flexibility is not paying for itself.
Cost per task varies widely on similar inputs. The loop is wandering.
The workflow has steps that never change the output. A critique pass that approves everything, or a router with one branch carrying nearly all the traffic. Measure each step's contribution and delete the ones that add nothing.
Signs you have under-built
One prompt is doing five jobs. Instructions interfere with one another, fixing one behaviour breaks another, and the prompt grows with every bug report. Split it into steps.
You keep adding branches for input types nobody predicted. The router has dozens of cases and new ones arrive weekly. The path may not be knowable in advance, and an agent over a well-defined set of tools may be simpler than the branching.
Humans are doing the orchestration. Someone runs a prompt, reads the output, decides which prompt to run next, and pastes results between them. If their decisions follow a pattern, that is a workflow. If they depend on judgment about intermediate results, that is a candidate for an agent.
The task needs information the prompt cannot hold. The model has to go and look things up, and what to look up depends on what it finds.
Guardrails when you do need an agent
An agent is software that takes actions chosen by a probabilistic component. Treat it with the same care as any automation that holds credentials.
- Tool allowlist. The agent gets the narrowest set of tools that accomplishes the task, each with the narrowest scope. Read-only by default. No general shell or unrestricted HTTP access unless the environment is disposable.
- Step, time, and cost budgets. Hard limits enforced by the orchestrating code, not requested in the prompt. When a budget is hit, the agent stops and reports what it has.
- Human approval for irreversible actions. Anything that sends, pays, deletes, publishes, or changes production waits for a person, who sees exactly what will be done.
- Sandboxing. Code execution and file operations happen in an isolated environment with no route to production data or internal networks beyond what the task requires.
- Untrusted input stays untrusted. Web pages, documents, and emails the agent reads can contain instructions aimed at it. Tool results are data, never commands, and the permission model must hold even if the model is fooled.
- Trace logging. Every model call, tool call, argument, and result, stored with a run identifier. This is both your debugging tool and your audit record; see audit trails for LLM systems.
- Deterministic checks on the result. Whatever the agent produces passes through validation that does not depend on a model before it is accepted.
How to evaluate each tier
Single prompt. A labelled set of inputs and expected outputs, scored automatically where the output is structured and by rubric where it is prose. Run it on every prompt or model change.
Workflow. The same, twice over: a test set per step, so you know which step regressed, and an end-to-end set, so you know whether the whole thing still works. Track cost and latency per step.
Agent. Score outcomes, not just final text: did the tests pass, was the right record found, was the correct root cause identified. That requires a reproducible environment the agent can run against, such as a fixture repository or a seeded database. Then add trajectory measures: steps taken, tools used, budget consumed, approvals requested, and any attempts at disallowed actions. Because paths vary, run each case several times and report the success rate rather than a single pass or fail.
The procedure, in short
- Write the task as a single prompt with the best context you can supply. Build a small test set and measure it.
- If it falls short, identify why. Missing context calls for retrieval. Too many jobs in one prompt calls for a chain. Distinct input types call for a router. Inconsistent quality calls for an evaluator loop or voting.
- Move to an agent only when the path cannot be specified in advance, and only after confirming that feedback is observable, actions are recoverable or gated, the latency is acceptable, and you can evaluate the result.
- Whatever tier you end on, review traces periodically and ask whether a simpler tier would now do. Models improve, and last year's workflow is sometimes this year's single prompt.
The question of whether a language model belongs in the system at all comes before any of this; we cover it in when LLMs are the right tool. If you would like a second opinion on an architecture you are planning, talk to us.
