When “Money Back” Means Refund: AI Topic Matching in the Wild

Every business should truly know its customers. That understanding begins where customers actually speak: in real conversations, with all their interruptions, restarts, filler words, and imperfect transcriptions.

That is the problem our new research examines. In Topic Matching in the Wild: Benchmark and Lessons from Real-World ASR Transcripts, accepted to the Workshop on Natural User-generated Text (WNUT) at EMNLP 2026, we study a foundational real-time AI task: recognizing when a customer’s live utterance is relevant to a predefined topic so an agent can receive the right guidance at the right moment.

The practical question is simple. If a customer says, “So can I, can I just get my money back?”, can a system recognize that they are asking for a refund—even if they never use the word “refund”? In production, that answer determines whether a useful coaching card appears in time to help a human agent move the conversation forward.

Our finding: the strongest approach was a lightweight LLM paired with a natural-language description of the topic. This approach outperformed both a production-style regex baseline and zero-shot sentence embeddings on human-annotated, real-world contact-center ASR transcripts.

The gap between clean benchmarks and live conversations

Many NLP benchmarks rely on written text. Customer conversations are different. They are spontaneous, fast-moving, and transcribed in real time. The transcript may contain repetitions, false starts, filler words, missing punctuation, and speech recognition errors, the exact conditions that make a real-time system useful, but hard to evaluate.

To study this setting, we created a human-annotated evaluation dataset from English customer-service calls across 11 companies and 173 topics. The data reflected the characteristics of spontaneous speech: 11.4% of utterances contained immediate word repetitions, 24.2% contained a filler or discourse markers. Because transcripts can contain sensitive customer information, we split the transcripts into short utterances, anonymize and exclude PII from the text prior to annotating the data in-house. 

Testing three ways to match a topic

We compared three matcher families on 2,655 human-labeled topic–utterance pairs:

• Regex matching, which is essentially topic keyword matching..

• Zero-shot sentence embeddings, which compare the semantic similarity of an utterance and a topic.

• LLM matchers, which make a focused YES/NO decision about whether an utterance matches a topic.

We also tested two ways to define a topic: a list of keyphrases, such as “refund,” “money back,” and “reimburse”; and a plain-language description, such as “The customer is asking to be refunded or to get their money back.”

That distinction proved decisive.

The right representation matters as much as the model

The highest-performing configuration, Gemini 3 Flash, reading a natural-language topic description, reached an F1 score of 0.847. That was 12.6 points higher than the regex baseline (0.721) and ahead of the best embedding approach (0.708).

The result was not simply that an LLM performed better. It was that the model and the topic representation had to work together. LLMs performed best with natural-language descriptions; embeddings performed best with keyphrase lists. Regex, by design, could only use keyphrases.

This is an important lesson for teams building real-time AI systems. The way a task is expressed to the system can be as consequential as the model chosen to perform it. A clear description gives an LLM a semantic target: it can recognize the intent behind a paraphrase or disfluent phrasing, rather than relying on an exact phrase match.

Smaller, faster models can be the practical choice

The research also produced a useful result for systems that operate under real-time constraints. The two top-performing systems were Flash-tier models; larger models did not improve the outcome.

Gemini 2.5 Flash-Lite reached an F1 of 0.833, within 0.02 of the best result, while measuring 346 milliseconds at the median per decision and an estimated marginal API cost of $0.010 per 1,000 decisions in our test. By comparison, Gemini 2.5 Pro was slower and more expensive without outperforming the lightweight models.

For teams operating live agent-assist workflows, that combination matters. Accuracy is essential, but it must arrive quickly enough to be useful and efficiently enough to be practical at scale.

Research grounded in how conversations actually happen

This work supports a broader principle behind Dialpad’s research: real-time AI should be evaluated in the conditions where people will rely on it. Businesses do not experience customers as polished benchmark examples. They experience them in live, imperfect, high-stakes conversations.

Dialpad is the AI platform for customer experience. Our research focuses on helping AI understand those real conversations with the context and rigor needed to support better outcomes while protecting sensitive information and keeping people in control.

Topic Matching in the Wild: Benchmark and Lessons from Real-World ASR Transcripts was accepted to WNUT 2026, held in conjunction with EMNLP 2026. The work is by Saman Rahbar, Xiliang Zhu, Irvin Cardoza, and David Rossouw of Dialpad.