This post continues the previous one: How to improve agentic APIs: expose a simple workflow API.

Summary

The previous post moved the workflow into deterministic code. The model still read each user turn and emitted structured tool calls. This post replaces that step with Jev, a TypeSafe model that answers classification questions. Jev interprets each user turn. Code applies the result to the booking state. The LLM has no tools and only crafts a conversational reply to speak to the user. The LLM is the "mouth" only.

Booking a tennis court needs free text fields such as a location and a contact name. Jev cannot generate text. It can only choose between options. I offer it the numbered words of the user's message and ask which word starts and which word ends the value. Code then copies that span verbatim.

The Jev agent and the simple workflow API agent were run on the same 11 scripted conversations. The Jev agent's speaker is now google/gemma-3-12b-it, a small non-reasoning model. The simple workflow API agent still uses deepseek/deepseek-v4-flash-20260731. Each passed 11 of 11. Total cost fell 90.8%, from $0.2866 to $0.0265. LLM input tokens fell 86.6%. Turn latency p50 fell 33.5%, from 2,974ms to 1,978ms. Turn latency p90 fell 45.4%, from 6,421ms to 3,508ms. Source: results/jev-gemma-vs-simple-workflow-api-deepseek.md.

Introduction

There is large potential to improve conversational agents with Jev. The previous experiment is worth revisiting to see how much further tokens, cost, latency and reliability can be improved.

As in the Jev Doom launch video, the key is to re-imagine the problem space as a classifier. Most of a booking conversation fits this shape. The user picks a surface, a duration or a time. Each is a choice from a small closed set.

Booking a tennis court is different from Doom in one way. It requires calling tools with free text fields. Jev supports neither tool calling nor generating free text values.

The question for this post is whether that limit can be overcome without compromising on latency, cost or reliability.

Jev Agent Design

The agent is in agents/jev_agent.py. The classification questions are built in jev/questions.py.

The tennis booking vocabulary is re-imagined as a set of classification questions that Jev can answer. Jev is the turn interpreter. The LLM is only the speaker.

The modified agent loop

A tool-calling agent runs a loop in which the LLM reads the conversation, decides whether to call a tool, receives the tool result and writes the reply. In the previous post's agent, that loop allowed up to one tool call per turn.

The Jev agent modifies that loop. Each turn runs these steps in a fixed order:

1. Jev interprets the user's turn by scoring the classification questions.

2. Code converts the answers to slot updates and applies them through the workflow engine.

3. The engine returns a payload describing the current state and the next question.

4. The LLM receives the payload and writes the reply.

Jev is the interpreter in step 1. The LLM only appears in step 4. There is exactly one Jev request and one speaker call per turn, and the LLM has no tools, so it cannot call anything or choose the next step.

How it works

Each user turn triggers one Jev request. The request carries the booking state, the last assistant message and the user's turn. It asks these questions:

• One Choice for the turn's act: answers_current, corrects_earlier, gives_later_info, asks_question, confirms, declines, restart_or_cancel, off_topic.

• One Choice per closed-vocabulary slot: surface, indoor or outdoor, skill level, duration, players, equipment rental and date. Each has a not_mentioned option.

• Two Choices per free-text slot (area and contact name) to find the span. See Text Capture Challenges.

What jev/apply.py does

jev/apply.py converts Jev's answers to slot updates. Answers below a 0.6 confidence threshold are not applied. They are passed to the speaker as ambiguities to clarify.

An ambiguity string carries the guessed value, its confidence, and the user's own words for that turn, not just a bare slot name. Without that context, the speaker had to replay the whole conversation to work out what an ambiguity was even about.

The workflow engine applies the updates deterministically. The LLM then receives the engine payload and writes the reply. It has no tools.

Text Capture Challenges

Jev answers with a choice. It does not generate text. Each slot that needs text is mapped to a choice.

• Court date: offer the next 14 dates, starting today, as choices. An ISO date typed by the user is also matched by regex and takes priority.

• Court time: once an availability search has run, offer every time slot of the courts found, plus not_mentioned. Before the search, this question is not asked.

• Court name: once an availability search has run, offer the names of the courts found, plus not_mentioned.

• Location: split the user's turn on whitespace into numbered words. The split is capped at 250 words. Ask Jev for the start word and the end word of the location. Code joins those words and strips edge punctuation.

• Contact name: the same start and end word method.

• Email: matched by regex in code. Jev is not used.

A span is only accepted if the start index is not after the end index. The confidence of a span is the lower of its two edges. Free text values are never generated. They are always copied from the user's own words.

Results

Jev's speaker is google/gemma-3-12b-it, a small non-reasoning model. The simple workflow API agent uses deepseek/deepseek-v4-flash-20260731 with default reasoning effort, its own default model. Both agents ran the same 11 scenarios. Source: results/jev-gemma-vs-simple-workflow-api-deepseek.md, the most recently generated comparison in the repo. Both agents passed 11/11 in this run, and the per-scenario detail shows reasoning tokens present on every simple_workflow_api call, consistent with its default reasoning effort setting.

The code and prompts are still under active iteration. Treat these numbers as a measurement of the current state, not a settled result.

Observations

• Jev's own cost, $0.0158, is 59.6% of the Jev agent's $0.0265 total, more than its speaker call ($0.0107).

• Input tokens per LLM call fell from 5,177 to 1,318. The speaker never sees tool schemas, conversation history for extraction, or anything beyond the current payload.

• LLM output tokens fell 58.3%, from 27,278 to 11,372. google/gemma-3-12b-it produces no reasoning tokens, unlike the reasoning-capable model behind simple_workflow_api.

• The speaker call itself (LLM latency, excluding Jev) was close between the two models: p50 1,564ms for Jev's speaker against 1,453ms for the tool-calling agent's calls, and p90 3,075ms against 3,897ms. Most of the turn latency gap comes from the tool-calling agent's extra round trips, up to 2 tool calls per turn, rather than from any single call being slower.

• Reliability was equal at 11/11 each. Neither agent had a failed scenario.

Learnings

• Moving turn interpretation to Jev cut LLM input tokens per call from 5,177 to 1,318, an 86.6% drop. The speaker never sees tool schemas, or anything beyond the instructions for the current conversation step.

• Small vertical domains (such as booking a tennis court) are a good fit for workflow -> fixed grammar conversion.

• Re-conceptualising conversational agents around function and specialisation unlocks necessary optimisations:

  • Turn interpreter, remodelled as a classification problem

  • State tracking, in deterministic code

  • Workflow state advancement, using a simple domain grammar in deterministic code, no tool calls

  • Natural conversation understanding and generation, with a small non-reasoning LLM

• Tool calling has always been an area of weakness with LLMs. Removing it altogether is appealing from a reliability standpoint, and the speaker's own tool-free design is part of why a small model can do its job: tool calling is itself a skill smaller models tend to be weak at.

• Running the 11 scenarios in parallel rather than sequentially was a big time saver, since each scenario is an independent scripted conversation.

Reasoning model learnings

• The speaker's reasoning, not the workflow logic, was the main cost and latency problem earlier in this project. Rather than continuing to manage that through prompt changes, swapping the speaker to a non-reasoning model removed the problem at the source. With the same prompt and payload shape, moving the speaker from deepseek/deepseek-v4-flash-20260731 to google/gemma-3-12b-it cut speaker output tokens from 18,243 to 11,372 (37.7% fewer) and total cost from $0.0617 to $0.0265 (57.1% cheaper). Turn latency p90 fell from 4,994ms to 3,508ms (29.8% lower), while turn latency p50 was roughly flat, 1,834ms against 1,978ms. Sources: results/jev-vs-simple-workflow-api-5-speaker-role-reframe.md and results/jev-gemma-vs-simple-workflow-api-deepseek.md.

• Sending the workflow payload to the speaker as a tool-result message, instead of as plain text in the user turn, cut its reasoning on its own, before the model swap. Tool-result messages are the channel a model is trained to quote as fact. Plain text is treated as prose to compose, which invited second-guessing of exact values such as prices and court names.

• Since the speaker's only job is to communicate the workflow state to the user, not to decide anything, a reasoning model was never necessary for it. google/gemma-3-12b-it, a 12B model, is confirmed to work at 11/11 with zero reasoning tokens.

• Some coding agents, such as Claude, are averse to reading reasoning traces. The traces give excellent insight into why LLM calls use more tokens and take longer. Explicitly asking the coding agent to read them is worth it when a latency or token result is surprising.

Future work

• Score conversation quality as part of the eval results, not just pass/fail and timing. Quality looked fine in manual testing, but was not tracked or measured.

• More evals to measure how well the Jev agent handles edge cases compared with an LLM.

• Repeat runs on more models, as in the previous post, to measure run-to-run variance. A single scenario is 9% of the pass rate.

• Stress-test free text capture: multi-word names, names that are also common words, values later in a long message, and messages over the 250 word cap. Add a check that a contact name span is not an email address.

• Ask only the questions relevant to the current node, to cut Jev input tokens.

• Tune the 0.6 confidence threshold against the rate of wrongly applied and wrongly skipped slots.

Code

All source code is in the GitHub repository. The Jev agent is in src/tennis_booking/agents/jev_agent.py. The question building is in src/tennis_booking/jev/questions.py.

This post and the code repository were written in collaboration with Claude.

Some background on where this comes from. I have spent 20+ years as a software developer, the last five building agentic AI products, and somewhere in there I picked up the kind of scar tissue that only comes from shipping a regression you should have caught. When I went looking for practical guidance on structuring evals for agentic systems, I found plenty of high-level takes and a crowd of paid products singing their own praises, but very few working playbooks. What follows is the approach that has worked for me, presented with runnable code. It is offered as a method to adopt and to argue with, in roughly equal measure.

The one-line change that broke everything

It started small. A prompt tweak: tightening a sentence, reordering two instructions, the kind of edit that looks harmless enough to skip a full review. It shipped. Nothing exploded, no alerts fired, no dashboard turned red.

A few days later, customer complaints started trickling in. The agent's outputs had shifted. Not broken exactly, just subtly wrong in a way that only showed up once real users leaned on it. We traced it back to that one-line change. It had nudged the model's behaviour just enough to break a downstream assumption nobody had written down anywhere. We found out from customers, not from a test run.

The lesson is one every backend engineer already knows: a prompt change is a code change. If you wouldn't ship a backend change without running your test suite, you shouldn't ship a prompt change without running yours. If you have customers, you need regression coverage.

Evals as unit tests, not next-day QA

A common pattern on teams building agents looks like this: evals exist, but they run as a separate process, often owned by a separate team, sometime after the code has shipped. This is an understandable place to end up. Eval tooling grew out of offline benchmarking, and it is natural to inherit that batch-oriented shape. But the ordering limits what the evals can catch. By the time the suite reports a regression, the regression has already reached users.

A unit test offers a different contract. It runs before merge, it blocks the build, and it gives a fast yes or no before the change touches anything real. There is no structural reason evals cannot meet the same bar. An eval can be a pytest test like any other, run the same way you would run any other suite:

uv run pytest -m eval_unit_test -v

The suite lives in the repo, it runs before merge, and a failure blocks the build. The rest of this article is about what those tests look like in practice.

The productionisation problem

Most agent tutorials stop the moment the demo works. Getting from "it works when I click through it" to "I'd trust this with real customers" is a different project entirely. It means:

• Replacing vibes with evals (the subject of this whole article)
• Putting prompts under version control and treating changes to them like code changes
• Adding observability, logging every tool call, every model response, latency, cost
• Building a test harness so you can iterate without fear of regression
• Deciding, concretely, what "good enough" means, and measuring it continuously

Evals are the load-bearing piece in that list. Without them, you're flying blind every time you change a prompt, upgrade a model, or extend the agent's capabilities. The move from POC to production is the move from "I'll know it when I see it" to "I have a suite that tells me within seconds whether this change is safe."

The industry has noticed, too. Job descriptions for agent engineers now routinely name evaluation frameworks and LLM-as-a-judge as first-class skills, listed alongside traditional unit testing, and they increasingly describe the work as non-deterministic engineering: handling probabilistic outputs, self-correction loops, and graceful degradation as a core discipline, not an edge case. One recent role description puts it plainly:

"Architect solutions that account for the probabilistic nature of agentic systems. Implement robust error handling, self-correction loops, and graceful degradation for when agents hallucinate, loop, or encounter edge cases."

Evals are how you practise that discipline day to day.

Patterns that hold up in practice

To make the patterns concrete, I built a small sandbox project: a tennis-court booking agent (a LangGraph agent with tools like get_user_location, find_nearby_courts, find_available_slots, and confirm_and_process_payment) with an eval suite alongside it. The full code is on GitHub: github.com/adam-lang2/tennis-booking. The snippets below come straight from it. They're illustrative, an example of what these patterns can look like rather than a production suite to copy wholesale, but every snippet runs, and the shapes translate directly to real systems.

1. Build a multi-turn test harness

Agents aren't stateless. They carry conversation history, tool call chains, and accumulated context across turns, and that's where most real failures live: three or five turns in, not in the first response. Your evals need to simulate realistic multi-turn exchanges, not just single prompt and response pairs. The sandbox does this with a ConversationBuilder: a fluent API for assembling a realistic history before you ever call the model.

# evals/convo_builder.py
class ConversationBuilder:
    def user(self, content: str) -> "ConversationBuilder":
        self._messages.append({"role": "user", "content": content})
        return self

    def assistant(self, content: str) -> "ConversationBuilder":
        self._messages.append({"role": "assistant", "content": content})
        return self

    def assistant_tool_calls(self, *calls: dict) -> "ConversationBuilder":
        formatted = [
            {"id": c["id"], "type": "function",
             "function": {"name": c["name"], "arguments": json.dumps(c["args"])}}
            for c in calls
        ]
        self._messages.append({"role": "assistant", "content": None, "tool_calls": formatted})
        return self

    def tool_result(self, tool_call_id: str, name: str, result) -> "ConversationBuilder":
        content = result if isinstance(result, str) else json.dumps(result, default=str)
        self._messages.append({"role": "tool", "tool_call_id": tool_call_id, "name": name, "content": content})
        return self

In a test, that reads as a script of exactly what already happened in the conversation, with mocked tool results standing in for the real API calls:

# evals/test_booking_evals.py
builder = (
    ConversationBuilder()
    .user("I'd like to book a tennis court.")
    .assistant("Where would you like to play?")
    .user("Near Central Park, New York City.")
    .assistant_tool_calls(
        tc("call_loc", "get_user_location", user_input="Central Park, New York City")
    )
    .tool_result("call_loc", "get_user_location", MOCK_LOCATION_RESULT)
    .assistant_tool_calls(
        tc("call_courts", "find_nearby_courts", location="Central Park, New York City")
    )
    .tool_result("call_courts", "find_nearby_courts", MOCK_COURTS)
)
builder.call_agent()

Everything up to .call_agent() is fixture: a scripted history, not a live run. Only the last step is real. Which brings up the second pattern.

2. Test one real step at a time, against a scripted history

You don't need to run the whole agent loop live to get a meaningful eval. The trick is to call the real model, with the real system prompt and the real tool bindings, for exactly one step, and stop before any tool actually executes. In the sandbox, that's what the agent caller does:

# evals/agent_caller.py
"""Helpers for calling the tennis booking agent LLM directly during eval tests.

Uses the production agent's model construction, system prompt, and tool
bindings (from tennis_booking.agent), so evals exercise one LLM step of the
real agent stack -- without the LangGraph tool-execution loop, which would
run real tools mid-eval and break the single-step eval design.
"""
from tennis_booking.agent import AGENT_TOOLS, SYSTEM_PROMPT, create_chat_model


def call_agent(messages: list[dict]) -> AgentResponse:
    """Call the booking agent model with the given conversation history.

    Prepends the production system prompt and applies the production tool
    bindings, then runs a single LLM step (no tool execution).
    """
    response = _get_bound_model().invoke(_to_langchain_messages(messages))
    return AgentResponse(response)

Note what's imported: AGENT_TOOLS, SYSTEM_PROMPT, and create_chat_model come from the agent package itself, not from a test double. The eval exercises the same prompt and tool bindings the agent actually uses, but no tool runs, so there are no side effects and no flakiness from live integrations. This is what makes it possible to test turn 4 of a conversation without replaying turns 1 to 3 through a live agent every time: you script the history you want as a fixture, and the eval answers exactly one question. What does the agent do right now, given this exact context?

3. Grade with an LLM judge when wording varies

Some agent behaviours can be checked with a plain assert: a routing decision, a required field in structured output. But a lot of what agents do has more than one correct answer. "Which tool did the agent call, and did the arguments make sense?" sounds deterministic until you realise the user's location might arrive as "Central Park," "near the park," or "Central Park, NYC," all correct, none an exact string match.

That's where an LLM judge comes in: a second model that reads the conversation and grades the agent's response against a written rubric. If you haven't used one before, here's what it looks like in the sandbox, using the deepeval library:

# evals/test_booking_evals.py
metric = ConversationalGEval(
    name="Calls location tool",
    criteria=(
        "PASS: The agent calls get_user_location with an argument referencing the user's stated location "
        "('Central Park', 'New York City', or similar).\n"
        "FAIL: The agent does not call get_user_location, or calls it without a location argument "
        "matching what the user mentioned."
    ),
    evaluation_params=[MultiTurnParams.TOOLS_CALLED],
    model=judge_model,
    threshold=0.7,
    async_mode=False,
)

Reading it piece by piece: criteria is the rubric, written as an explicit PASS/FAIL binary rather than "is this response good?", so the judge has almost no room for interpretation. evaluation_params tells the judge what evidence to look at (here, the tools the agent called; other tests also include the message content). model is the judge itself, a separate model from the agent, more on that below. And threshold converts the judge's score into a pass/fail that pytest understands, so a failing rubric fails the test like any other assertion.

The rule of thumb: write the tightest check the actual variability of your input allows. If there's exactly one correct answer, use a plain assert. It's faster, cheaper, and can't be wrong. If correct answers legitimately vary in wording, use a judge with a strict binary rubric. What you want to avoid is the vague middle: a fuzzy rubric ("did the agent respond helpfully?") produces noisy, inconsistent scores and catches nothing.

4. Rerun for reliability, and know which kind of rerun you need

LLMs are non-deterministic, so how much a single run tells you depends on what you're testing. There are two distinct questions:

• Can the agent do this at all? (capability: the agent might phrase things differently across runs, and passing once proves the capability exists)
• Does the agent do this every single time? (reliability: guardrails and critical behaviours, where a single failure is a real bug)

Mixing them up gives you the wrong signal in both directions. Treating a safety guardrail as "only needs to pass once" lets a real regression through, while demanding "must pass every time" from ordinary phrasing variance makes your suite flaky for no reason.

Both are expressible with plain pytest markers, the same tools you already use for flaky integration tests:

# evals/test_booking_evals.py
# For invariant tests, re-run them N times to guard against false negatives.
# All have to pass.
INVARIANT_REPEATS = 3

# For trajectory tests, re-run them N times to allow for non-deterministic behaviours.
# Only one has to pass.
FLAKY_RERUNS = 3

A guardrail like test_agent_refuses_to_book_without_email gets @pytest.mark.repeat(INVARIANT_REPEATS), meaning all three runs must pass. A capability check like test_agent_calls_location_tool_after_user_gives_location gets @pytest.mark.flaky(reruns=FLAKY_RERUNS), where one success is enough. The flaky treatment is particularly useful when the LLM has multiple correct ways to reach a result: the agent might call the location tool immediately, or ask a clarifying question first and call it next turn. Both are legitimate paths to the same outcome, and a test that demands one specific path on every run will fail on behaviour that isn't actually wrong. The practice worth adopting is the distinction itself: decide per-test whether you're measuring capability or reliability, and rerun accordingly.

One caveat that saves a lot of API spend: reruns are for behaviour that actually varies. In my experience, many regressions, once you've captured the exact conversation state that triggers them, reproduce 100% of the time. For those, a single run gives you a clear signal, and rerunning just costs money. Save the repeats for behaviour that's legitimately probabilistic.

5. Write negative tests, not just happy paths

Two of the most valuable tests in the sandbox suite don't check that the agent does the right thing. They check that it doesn't do the wrong thing when it's tempting to. One confirms the agent asks for an email before booking rather than silently proceeding:

# evals/test_booking_evals.py
metric = ConversationalGEval(
    name="Asks for email before booking",
    criteria=(
        "PASS: The agent asks the user for their email address before proceeding. "
        "It must NOT call confirm_and_process_payment.\n"
        "FAIL: The agent calls confirm_and_process_payment without having received an email "
        "address, or proceeds with booking without requesting the missing information."
    ),
    evaluation_params=[MultiTurnParams.CONTENT, MultiTurnParams.TOOLS_CALLED],
    model=judge_model,
    threshold=0.7,
    async_mode=False,
)

The other confirms the agent doesn't fabricate availability when find_available_slots comes back empty:

# evals/test_booking_evals.py
.tool_result("call_slots", "find_available_slots", [])  # no slots available
...
criteria=(
    "PASS: The agent informs the user that no slots are available on the requested date "
    "and suggests an alternative action...\n"
    "FAIL: The agent fabricates slot availability, calls confirm_and_process_payment, "
    "or gives no useful response to the lack of availability."
),

Neither of these would ever show up in a happy-path demo. They exist because someone thought about what the agent could get away with, not just what it's supposed to do.

6. Don't let the agent grade its own homework

If your production model and your judge come from the same model family, the judge tends to share the same blind spots as the thing it's grading. The sandbox bakes this into its judge fixture, with the reasoning written directly above it:

# evals/conftest.py
# The judge MUST be a different model family from the agent under test.
# Grading with the same model produces correlated errors -- the judge will
# rubber-stamp the agent's blind spots rather than catch them.
_DEFAULT_JUDGE_MODEL = "google/gemini-2.5-flash"


@pytest.fixture(scope="session")
def judge_model() -> OpenRouterModel:
    """GEval judge model. Uses a different model family from the agent -- see comment above."""
    api_key = os.environ["OPENROUTER_API_KEY"]
    return OpenRouterModel(model=_judge_model_name(), api_key=api_key)

Note the judge is also a cheap, fast model, not the biggest one available. A judge call is just an API call; at fractions of a cent per case, a 50-case suite costs pennies per run. That's CI-budget money, and it's what makes judge evals viable in the local dev loop, not just in a nightly pipeline. Routing through OpenRouter means swapping judge models is an env var change, not a code change, which is useful for checking that your suite isn't calibrated to one judge's particular quirks.

When production breaks, write the eval first

Here's where the unit-test discipline pays for itself twice. When a bug is reported from production, the first move is to write a failing eval that reproduces it, then fix the prompt until it passes reliably, not just once.

This is the same reflex a good engineer already has for ordinary bugs: reproduce as a failing test, fix, keep the test forever. Applied to agents, it turns the scariest class of bug, "the model sometimes does something weird," into a routine workflow. You capture the exact conversation state that triggers the failure as a ConversationBuilder fixture, write the rubric for what should have happened, watch it fail, and iterate on the prompt until it passes. The eval then stays in the suite permanently, so that specific failure can never quietly come back.

The infrastructure from the patterns above (conversation builder, single-step caller, judge fixture) is what makes this a routine step rather than a scramble: you're plugging a new case into an existing harness, not starting from scratch per incident. Without that harness, "reproduce the bug" means clicking through a chat UI hoping to trigger it again. With it, it's twenty lines of fixture.

A human still has to read the evals

One thing that's easy to lose in all this automation: every eval encodes a human decision about what "correct" means. The rubric in a judge eval, the assertion in a deterministic one, the choice of which behaviours count as invariants. Those are judgment calls, and they should be reviewed with the same care as the code they protect.

This matters more, not less, as more of the codebase becomes AI-generated. When large parts of a system are written, and sometimes never closely read, by a model, the eval suite is one of the few places where a human is still explicitly deciding what good looks like, writing it down, and checking it continuously. Review every eval that enters the suite. Read the judge's reasoning when a case fails, not just the pass/fail. An eval nobody has scrutinised is a test of nothing.

A working vocabulary

Application metric: your top-line pass rate across the whole eval suite. Good for tracking regression over time; too coarse on its own to debug an individual failure.

Per-step (or per-turn) eval: an eval that calls the real agent for a single step against a scripted prior conversation, rather than judging only a full end-to-end run.

Multi-turn eval: an eval spanning a full conversation, not a single prompt-response pair. Catches context drift and compounding errors that single-turn evals miss.

Model-graded eval: pass/fail determined by an LLM judge against a rubric. Necessary for anything with legitimate variation in correct phrasing.

Programmatically graded eval: pass/fail determined by code: exact match, schema validation, regex. Fast, cheap, fully deterministic, and the right default whenever there's no legitimate variation in what "correct" looks like.

Invariant test: a test that must pass on every rerun. Used for guardrails, safety behaviours, and critical application behaviour that can never regress even once.

Capability (trajectory) test: a test that only needs to pass once across several reruns, because minor variation across runs is expected and acceptable.

Prod-regression eval: written after a production failure to make sure that specific incident never regresses. A healthy suite has both prod-regression evals (closing gaps you've been burned by) and code-first evals (anticipating gaps before they open).

Start small, grow with every feature

You don't need a platform, a vendor, or a perfect taxonomy of eval types to start. What you need is: a way to script a conversation, a way to call your real agent for one step against it, and a judge that isn't the same model you're testing.

There's a compounding effect here worth naming. Each eval you add becomes a primitive: a building block that pins down one behaviour and stays pinned. A suite of those primitives is a stable base for iteration. You can rework a prompt, swap a model, or bolt on a new tool, and know within minutes which existing behaviours held. That stability is what unlocks new features. Without it, every change reopens every old question, and the cost of touching the system grows with its size instead of shrinking.

Then make it a habit, not a project. Every new feature lands with its evals in the same PR, the same way a new endpoint lands with its unit tests. Every prompt change runs the suite before merge. Every production bug becomes a new eval before it becomes a fix. You never stop to "do evals" as a phase; the suite grows in parallel with the agent, one case at a time, and each case is small: a scripted history, one real step, one rubric.

Have a better pattern, a framework you'd recommend, or a war story of your own? I'd genuinely like to hear it. This is meant to start a conversation, not close one.

Summary

Conversational agents are often built by writing a workflow into the system prompt and letting the model work out which step it is on. The common answer to the reliability problems that follow is to move the workflow state out of the prompt and into an explicit graph, with LLM calls driving the transitions. That adds a second round of orchestration to every turn. I ask whether the step tracking and the transitions can be moved into deterministic code entirely, leaving the model only the part it is good at, talking to the user, and what that costs in tokens and latency. The same 16-step tennis-court booking workflow is implemented three ways: a ReAct agent that infers its own position from the transcript, a prompt-graph agent that keeps state server-side and advances it with an LLM call each turn, and a simple workflow API agent whose single tool takes {slot, value} updates and is backed by a LangGraph StateGraph containing no LLM calls. Each is run against 11 scripted conversations, 10 of which contain corrections, multi-answer messages or off-topic turns, on two models, twice over: 132 conversations in total.

The three architectures do not hand the model the same tools, so the comparison is on tool call correctness rather than tool choice. Scoring reads the arguments that reach search_availability and book_court. It does not matter whether the model emitted those calls itself or the workflow engine made them internally. Every agent records them in the same tool call log, so one scoring function runs over all three unmodified. The chat text is not scored.

The simple workflow API agent passed 43 of 44, the prompt-graph agent 40, and the ReAct agent 32. The simple workflow API agent was the only architecture to complete the workflow reliably on both models, so the other agents' failures track the architecture rather than the model. That reliability costs about 2× the ReAct agent's tokens, while the prompt-graph agent costs 2.4–2.8×, because every turn adds an orchestration round trip to a stateless API that resends the whole history. A one-line addition to the ReAct agent's system prompt, written to address its dominant failure, did not recover it: re-running the complete suite with that line left it at 15 of 22, inside the range the unmodified agent had already produced.

Introduction

I've spent the last six months as an AI engineer building customer-facing voice agents.

As an AI developer there's pressure to deliver reliable agents quickly. At "some scale", the standard approaches hit an invisible wall: the model can't cope with the complexity, and its prose-based rules start to blur into each other.

LangGraph calls out this limitation and recommends mixing deterministic and LLM steps. But LLM steps add complexity and latency, which are arguably a poor match for conversational settings.

This post tests a different approach: selectively trading probability for determinism where it's most needed. The more consistent the workflow path, the more a static workflow becomes a worthwhile, even necessary, optimisation. The objectives behind that trade off are the ordinary production ones: low latency and token cost, reliability with small and fast models, deterministic primitives, smaller prompts in place of prose based rules, agents decoupled from their tools, and no loss of conversational flexibility, so a user can still answer three questions at once or change their mind two steps later.

So I built the same agent with three different architectures and measured their performance.

Repository: github.com/adam-lang2/prompt-chain-vs-workflow-steps-vs-simple-workflow-api

Contributions

• A benchmark of 11 scripted conversations over one 16-step booking workflow. It runs against three agent architectures on two models, twice: 132 conversations in total. All three agents share one workflow definition and one scoring function. Scoring inspects the arguments that reach search_availability and book_court, not the chat text. The architecture is the only thing that differs between cells. The code, the scenarios and both result files are in the repository.

• Evidence that the architecture, not the model, decides whether the workflow completes. The simple workflow API agent passed 43 of 44 conversations. The prompt-graph agent passed 40 and the ReAct agent 32, on the same models and the same scenarios. In 11 of the ReAct agent's 12 failures, search_availability was never called at all.

• A cost comparison in which the cheapest architecture is the least reliable. The ReAct agent is cheapest in every cell. The simple workflow API agent costs about 2× it on both models. The prompt-graph agent costs 2.4–2.8× on DeepSeek V4 Flash and 9.3–11.9× on Luna. Its input tokens grow faster than its request count. On Luna it made 2.8–3.0× the requests but used 9.7–12.6× the input tokens. The Chat Completions API is stateless, so every extra round trip resends the whole conversation history.

• Reliability that doesn't require the more capable model. The simple workflow API agent passed 22 of 22 on the weak model. On that model the ReAct agent lost 2 passes and the prompt-graph agent lost 4, relative to their scores on the strong one.

• A prompt-level mitigation, tested and negative. The ReAct agent's dominant failure is never calling search_availability. I added one line to its system prompt aimed at that behaviour and re-ran the complete suite. It scored 15 of 22, inside the 15–17 range it had already produced without the line. Five of its seven remaining failures are still that same behaviour.

Related work

Workflow orchestration. LangGraph models a workflow as nodes that do the work and edges that decide what runs next.

Constrained generation. Structured outputs and grammar-guided decoding constrain the form of what a model emits. Willard and Louf give the finite-state formulation, implemented in Outlines. The guarantee is text that parses against a regular expression, a grammar, or a JSON schema.

Slot filling and frame-based dialogue. The {slot, value} grammar is not new. It is the frame of GUS (Bobrow et al., Artificial Intelligence 8(2), 1977), a travel-booking dialogue system. Its frames held slots the user could fill in any order, under mixed initiative, several at a time.

In all of this work the thing varied is the model, or the format of what it emits, or the prompt given to a fixed agent. Here the model, the workflow, the scenarios and the scoring are held fixed, and what varies is how much of the workflow the model is responsible for.

Sample task: booking a tennis court

The experiment:

1. Task: book a tennis court through a conversation. The workflow collects the user's preferences, searches court availability, lets the user pick a time, collects contact details, confirms, and books. It's 16 steps, defined once in code and shared by every agent.

2. Agents: three agents implement that same workflow in three different ways (next section).

3. Evals: every agent runs the same 11 scripted conversations. One answers every question in order. The other 10 are messy: corrections, several answers in one message, off-topic questions, a malformed email.

4. Scoring: a conversation passes if the arguments that reach search_availability and book_court match what the user asked for. The chat text isn't scored.

5. Models: each agent runs on a weak model, GPT-5.6 Luna with reasoning off, and a strong model, DeepSeek V4 Flash.

6. Runs: the full suite (3 agents × 2 models × 11 conversations) was run twice unmodified, giving 132 conversations, and then a third time after one line was added to the ReAct agent's system prompt, giving a further 66. The third run is reported separately, under Mitigation.

The three agents

1. ReAct agent

One static system prompt lists all 16 steps. On every turn the model re-reads the transcript and works out which step it's on. Nothing tracks progress outside the model.

Tools: The agent is given all the tools (search, book, confirm) to effect a successful booking.

Code: agents/react_agent.py

Expected problems:

• At a certain scale it forgets which step it's up to. It has no "current step" pointer, so it has to work out its position for every response, and depending on the conversation content (and length), token weightings can bias it towards skipping or repeating steps.

• Workflow steps often contain rules written in prose, which can overlap with other rules in unexpected ways.

• How much weight the model gives a step depends on where the step sits in the prompt, and on other factors that affect the model's attention emphasis.

• Prompt bloat: the prompt carries every workflow step and every tool.

• As steps change and more are added, the agent gets locked into an unmanageable structure.

2. Prompt-graph agent

This is the standard LangGraph pattern: workflow state lives on the server, and LLM calls drive the workflow forward. The system prompt describes the workflow at a high level. Every turn, the model calls get_next_step() with whatever the user just said, and gets back one instruction: a question to ask, or a tool to call. Progress is stored server-side, keyed by conversation ID.

Tools: The agent is given the standard booking tools (search, book, confirm), plus get_next_step().

Code: agents/prompt_chain_agent.py

Expected problems:

• Double the token cost and latency: one LLM call to talk to the user, plus another round of LLM calls to orchestrate the workflow.

• It's poor for tokens and latency because every turn makes an extra tool call to advance the workflow, and every tool still has to be in the prompt.

• Models with weaker tool calling get less reliable as more tools are added. Can be mitigated by scoping tools to the current step only.

3. Simple workflow API agent

The proposed solution:

• Workflow structures are usually known ahead of time, encode them into a simple API that an agent tool can follow.

• Even small models can write good-enough tool payloads for a simple grammar based API.

• It takes a conceptual leap: you have to define the workflow in terms of a simple grammar that even a small model can follow.

The model has one tool, book_tennis_court_with_grammar. It takes a list of {slot, value} updates. Any slot can be sent at any time, in any combination: a new answer, a correction, or an answer to a step the conversation hasn't reached yet. A user who opens with where, when, and how long produces one call:

{"updates": [
  {"slot": "area", "value": "Discovery Park"},
  {"slot": "date", "value": "2026-09-23"},
  {"slot": "duration_minutes", "value": 120}
]}

Behind the tool is a LangGraph StateGraph with no LLM calls in it. It validates each slot, works out the next step, and returns the next instruction for the model to put to the user. When the workflow settles on the search or booking step, the engine calls search_availability and book_court itself. Those are the same two implementations the other two agents hand to the model as tools. The model in this architecture is never given them and never calls them. The engine records each internal call it makes, so scoring reads the same two sets of arguments for all three agents.

Code: agents/simple_workflow_api_agent.py, workflow_engine/

Results

Raw output from the two unmodified runs, as generated by tennis-compare. Both use the three agents exactly as described above; the third run, with one line added to the ReAct agent's prompt, is reported under Mitigation. Totals and averages are over the 11 conversations in each cell. "Latency" is per LLM call; "turn latency" is per user turn (all LLM calls and tool calls needed to answer one user message).

Weak = openai/gpt-5.6-luna, reasoning effort none. Strong = deepseek/deepseek-v4-flash-20260731.

Run 1: tokens and cost

Run 1: latency

Run 2: latency

Full reports, including every mismatched argument: results/. The raw files use the code's agent ids: prompt_chain for the prompt-graph agent, simple_workflow_api, and workflow (run 1), workflow_steps (run 2) or react (run 3) for the ReAct agent.

Observations

Both runs combined. Ranges span run 1 and run 2. "vs ReAct" is a multiple of the ReAct agent's figure on the same model and run. "LLM reqs/conv" is LLM requests per conversation. "Turn p50" and "Turn p90" are turn latency, meaning all the LLM and tool calls needed to answer one user message.

• Pass rate. Simple workflow API passed 43/44, and its one failure was a paraphrased area value (near Golden Gate Park for Golden Gate Park). ReAct passed 32/44: in 11 of its 12 failures search_availability was never called, so the conversation never reached the search step, and it failed messy_all_at_once_opener (the first message answers every search question plus name and email) in all 4 of its runs. Prompt-graph passed 40/44: all 4 failures were on the weak model and searched with surface='any' after the user asked for hard courts, including the in-order scenario, where the user's third message is "I prefer hard courts." Prompt-graph tracks the step server-side, but the model still writes the search arguments itself. Run-to-run variance is large for the two model-driven workflows (ReAct on the strong model went from 7/11 to 10/11, prompt-graph on the weak model from 10/11 to 8/11). Simple workflow API scored 10–11/11 in every cell.

• Token use. ReAct is the cheapest agent and the least reliable. It makes the fewest LLM requests because it only calls tools to search, book, and confirm. Simple workflow API costs about 2× ReAct on both models, and that multiple holds across runs: it calls book_tennis_court_with_grammar on most turns, and each tool call adds another LLM request to phrase the reply. Prompt-graph calls get_next_step every turn and then whatever tool that step names, and on the weak model it made up to 8 tool calls in a single turn. The Chat Completions API is stateless, so every extra round trip resends the whole conversation history. That's why prompt-graph's input tokens grow faster than its request count: on the weak model it made about 3× ReAct's requests but used 10–13× its input tokens. Prompt-graph cost 1.3–1.4× as much as simple workflow API on the strong model, and 4.6–5.7× on the weak model.

• Latency. Turn latency follows the number of LLM calls a turn needs. ReAct has the lowest p50 on both models. Prompt-graph on the weak model has a p90 of 10–14 seconds. Simple workflow API on the weak model has about the same p90 as ReAct (3.3–3.5s vs 3.2–3.5s). On the strong model its p90 is higher (5.0–5.9s vs 3.3–3.4s), because most of its turns take two calls and the strong model's per-call p90 on this agent is about 2.5s.

• Weak vs strong model. The weak model made both model-driven workflows less reliable: ReAct went from 17/22 on the strong model to 15/22 on the weak one, and prompt-graph from 22/22 to 18/22. Simple workflow API passed 22/22 on the weak model. The weak model was cheaper for ReAct and simple workflow API (0.75–0.87× the strong model's cost), because it wrote far fewer output tokens per call (45–53 vs 132–201) even at twice the output price. For prompt-graph it was 2.85–3.74× more expensive, because it made more calls (41–44 vs 30 per conversation) with larger inputs (11,270–13,380 vs 5,103–5,303 average input tokens per call).

Mitigation: one line in the prompt

The ReAct agent's failures are concentrated in a single behaviour: in 11 of its 12 failures across runs 1 and 2, search_availability was never called, so the conversation never reached the search step. If that is a prompt problem rather than an architectural one, the cheapest possible fix ought to move it. One line was added to the ReAct agent's system prompt, and nothing else was changed:

When a single message answers several steps, record them all before deciding what to ask next.

The wording is deliberately generic. A line naming the search step, such as "call search_availability as soon as you have everything it needs", would target the exact scenarios already known to fail. That would measure my knowledge of those failures rather than the value of the fix. The complete suite was then re-run: same 11 scenarios, same two models, same scoring, same decoding settings, with the other two agents untouched so that they act as a control on the run itself.

Run 3 (added line in the ReAct agent's prompt only): tokens and cost

Run 3: latency

The added line did not recover the failures. With it, the ReAct agent scored 15/22, inside the 15–17/22 range it had already produced without it and below its run 2 score. Five of its seven remaining failures are still search_availability never being called. messy_all_at_once_opener, the dense opening message the line was written for, failed on both models again: 6 failures out of 6 attempts across the three runs, with and without the line.

Scope: one prompt change, one run, 11 conversations per cell. This does not show that no prompt can fix the ReAct agent. It shows that the obvious prompt fix, aimed directly at the dominant failure, did not fix it here, and that the obvious "levers" for this agent architecture are unreliable.

Learnings

You can engineer a better agentic API. The simple workflow API agent passed 43 of 44 conversations, and the one failure was a paraphrased area value. It cost about three-quarters as much as the prompt-graph agent on the strong model, and about a fifth as much on the weak model.

• Most conversational agent platforms aren't set up to follow this pattern natively. They lead to higher token use and higher latency instead.

• Tool APIs are typically pre-AI legacy code, not adapted to LLM language and grammar capabilities.

• Redesigning agentic tool APIs around simple grammars works well, even for weak models: the simple workflow API agent passed 22/22 on the weak model.

• Being deliberate about when LLM calls are required, and aiming them at LLM strengths such as conversing with users, is key to achieving quality AI systems.

• Being more willing to build out deterministic primitives can pay off a lot. Here, the staleness and validation rules are typed slot validators with unit tests, instead of prose in a prompt.

• A prompt-level fix aimed squarely at the ReAct agent's dominant failure did not improve performance. That is one wording and one run, so it doesn't show that no prompt could. It does show that prompt-shaped fixes for structural problems often fail.

Future work

Future lines of investigation, out of scope in the current work.

• Allow LLM calls inside the workflow engine for complex nodes, or in error scenarios as a graceful fallback, using LangGraph LLM steps.

• Surface multiple steps in a single response, to batch steps and further cut latency and token cost.

• Allow forward and backward step skipping, to improve conversation flow.

• Trim early tool calls from the conversation history to minimise context bloat.

Code

All source code, the scripted conversations, and all three result files: github.com/adam-lang2/prompt-chain-vs-workflow-steps-vs-simple-workflow-api

To reproduce both models in one run (needs an OpenRouter key):

uv run tennis-compare \
  --model openai/gpt-5.6-luna:none \
  --model deepseek/deepseek-v4-flash-20260731 \
  --format markdown

Appendix: every failure

Every conversation that failed scoring in runs 1 and 2, with what the scoring found. Run 3’s failures are described under Mitigation.