New Dialpad research shows why an agent’s next-turn score is not enough to prove it can complete a customer workflow.
Every business should truly know its customers. That requires AI agents that can carry context forward, use tools correctly, recover from mistakes, and complete work, not simply produce a plausible response to the next turn.
But much of today’s agent evaluation stops too early. A model is shown the ideal conversation history and scored on whether its next action matches a reference. That can be useful. It is not the same as asking whether the agent can navigate an entire workflow with its own evolving interaction history.
Our new paper, When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success, accepted to the REALM workshop at EMNLP 2 on 026, investigates that gap in multi-turn customer-support workflows.
A strong next turn can hide a weak workflow
We evaluated pre-SFT and supervised fine-tuned Qwen3 and Gemma 3 models across four model sizes. Under conventional gold-history evaluation, SFT improved text-turn success for every model. At first glance, the results looked encouraging.
But the picture changed when the agents had to execute full workflows autonomously using their own interaction history. None of the four SFT models successfully completed a tool-requiring workflow under the study’s strict trajectory-completion measure. The highest workflow success rate was 10.4%.
This does not mean turn-level evaluation is useless. It means it is incomplete. An agent can become better at predicting the next action in an idealized context without becoming better at maintaining state, choosing the right tool at the right time, handling the consequences of prior actions, and completing the customer’s task.
The metric should match the work
For customer-facing agents, a well-formed response is not the outcome. Resolution is. The relevant question is whether the system can carry a workflow through to the intended end state, with the right decisions and tool execution along the way.
Our results point to a more complete evaluation stack:
Text quality: Does the agent communicate clearly and appropriately?
Local action correctness: Is its next decision sensible in context?
Tool execution: Does it call the right tool with the right arguments?
Workflow completion: Can it complete the task when operating on its own history?
Reporting all four prevents a polished turn-level metric from standing in for the outcome that matters most.
Why this matters for production AI
Real customer workflows are stateful. One decision changes what information is available next. One failed tool call can alter the trajectory. A workflow that looks easy from a gold history can become difficult when the agent must live with its own decisions.
That is why evaluation needs to reflect the conditions of deployment. A system may look capable in an offline benchmark while still failing to complete work autonomously. Teams building agents need evidence not only that the model can identify an appropriate local action, but that it can sustain progress through an entire task.
Dialpad is the AI platform for customer experience. We believe intelligence creates value only when it changes what happens next. This research reinforces a related principle for agent evaluation: local intelligence is not enough if it does not lead to reliable end-to-end execution.
The study is intentionally limited to two model families and one proprietary customer-support domain, and its closed-loop replay requires exact reference tool calls. Valid alternatives may therefore be rejected, and additional domains and interactive environments should be studied. The central result remains important: next-turn success is not a reliable proxy for autonomous workflow success.
When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success was accepted to the REALM workshop at EMNLP 2026. The paper is authored by Gundeep Singh and colleagues at Dialpad.