I tried to replace an LLM with a decision model for routing phone calls

Two weeks with Cloudflare’s Clef family, TypeSafe’s Jev and GPT on German customer-service calls, including everything that went wrong along the way.

---

Every contact center that experiments with voice AI runs into the same unglamorous problem. Before an agent can help anyone, it has to work out what the caller wants, whether that is a blocked card, a travel booking, a burst pipe or a wrong number.

In our voice prototype this decision lives in a prompt, where a large language model reads the caller’s first sentence and picks a category. It works, but it takes seconds, it gets expensive at scale, and the confidence an LLM reports about its own answer rarely means much. When a new kind of model started getting attention this autumn, I wanted to know whether it could do this job better.

A classifier on steroids?

TypeSafe introduced a model called Jev together with an API they call “System One”. You send the model a state, such as a piece of text, and a set of typed questions that ask it to pick an option, rate something on a scale or answer yes or no. It answers all of them in a single forward pass and returns a probability for every option without generating any text.

On October 1, Cloudflare released Clef (27B) and Clef-flash (9B) with the same API, as open weights and hosted on Workers AI. The announcement quoted a median latency of 39 milliseconds for the small model, and for a voice application, where every few hundred milliseconds of silence feels awkward, that number was the reason I started this experiment.

Building something worth measuring

The task is to route a caller’s first sentence to one of eleven intents, from card problems and home emergencies to travel, cancellations and insurance claims, plus a bucket for anything that has nothing to do with us.

Sending real conversations from people who call our hotline to vendors whose models were a few days old was out of the question, so the data had to be synthetic. To make it sound like a phone call, an LLM wrote German first sentences for ten rotating personas, from the elderly lady who starts with the whole backstory to the man calling from his car. A text-to-speech voice read each sentence, an 8 kHz codec turned it into phone-line audio, and Whisper transcribed the result. The models therefore saw what the speech recognition heard, which led to my favourite example of the whole project: a caller who wanted to widerrufen (revoke) a contract became a caller who wanted to wieder rufen (call again).

Before running anything important, I wrote down what success would look like: accuracy within three points of the best LLM, a confidence signal I could trust, and a latency that suits a voice conversation.

When everything scores 99%, the test is the problem

The first full run made the LLMs look excellent. GPT 4.1 mini reached 99.1% and GPT 5.4 mini reached 98.6%, and a result like that should make you suspicious rather than happy. A GPT model had written the test sentences, so they were textbook examples of each category, which is exactly what LLMs from the same family recognise best.

I built a second, “hard” set with a different generator model and asked it to be messy, with callers who ramble, describe their problem without naming it or mention two issues at once. Everyone collapsed on it, the LLMs to 73–77% and Clef along with them.

My first explanation was that my generated labels had to be wrong, and I wrote “probably label noise” into my notes. A domain expert from our contact center then reviewed every test label and confirmed 410 of 417, corrected four and marked three as ambiguous. The hard set was simply hard, and my theory had been a comfortable excuse that I am glad someone checked before I published it.

The night GPT 4.1 mini dropped to 48%

At one point GPT 4.1 mini fell from 99% to 48% between two runs, and for about ten minutes I believed I had found something interesting.

What I had found was rate limiting. Azure had throttled half of the requests, my client hid the HTTP status code, the retry logic never started, and every failed call counted as a wrong answer. In the same week, the evaluation framework I used handed a callback a different object than its type declared, so one run recorded every row as an error while the built-in scorer reported 100%. Since then I look at the error column of a benchmark before the accuracy column, because benchmarks fail silently far more often than models do.

Two sentences mattered more than any model

With trustworthy infrastructure, one weakness showed up in every decision model. They struggled with the “nothing to do with us” bucket, which Clef recognised in 65% of cases and Clef-flash in only 20%, usually confusing it with “the bank or a partner is responsible”.

Those two categories are close in meaning, and my descriptions had not drawn a clear line between them. I rewrote both, so that the first one only covers the caller’s own bank account or card and our program partners, and the second covers everything unrelated to us or the bank. That change lifted Clef-flash’s recognition of out-of-scope calls from 20% to 94% and added one to eight points of overall accuracy to every model, the LLMs included. None of the model swaps I tried came close to that effect.

A second idea was less universal. I added a yes-or-no question to the same request, asking whether the call concerns our program or the bank at all, and used it as a gate. For Clef it worked well enough to match the best LLM on the development set, while for Jev and Clef-flash it fired far too often and pushed valid calls into the out-of-scope bucket.

If you are about to compare models for a routing task, spend an afternoon with the people who answer those calls first, because your label definitions are a product decision.

Confidence turned out to be the real feature

A voice agent does not have to route every call on its own. When the model is unsure, the agent can ask whether the caller means the card or the travel booking, which costs a few seconds, whereas a wrong transfer annoys the caller and costs the contact center real time. The useful question is therefore how many calls a model can route automatically while staying 98% precise.

Decision models suit this question because their probabilities are real distributions. Clef-flash sent the wieder rufen call to the wrong team with 94% confidence, while Jev answered “other” with 59% and would have triggered a clarifying question instead. GPT 4.1 mini was very accurate, but its self-reported confidence was so coarse that no threshold brought it to 98% precision.

Halfway through I also had to admit that my own criterion was too weak. Requiring 70% of calls to be routable at 95% precision let Clef and Jev pass without abstaining on a single call, so I tightened it to 85% at 98% precision. Because that looks a lot like moving the goalposts, I wrote it down before generating new data and applied it only to a held-out test set that my tuning had never touched.

Latency, or where the hype met the network cable

Cloudflare quoted 39 ms for Clef-flash and 209 ms for Clef, while my first measurements from a laptop in Germany showed medians between 400 and 600 ms and tails beyond a second.

My connection to Cloudflare was genuinely poor that day, so I suspected the network and deployed a tiny Cloudflare Worker that reports how long the model call takes on Cloudflare’s side. The network added 20 to 50 ms and the rest was model time. Clef-flash showed a strange split, with a few calls at about 190 ms, most between 650 and 1,100 ms and outliers up to 3.5 seconds, which looks like queueing on a model everyone was trying out that week.

I made a mistake here as well. When latencies looked much better a day later, I wrote that this confirmed load-dependent variance at Cloudflare, before remembering that I had been working from a different location that day and could not separate location from load. I corrected the note, but claims like that are exactly what ends up being repeated as fact.

For a voice agent the practical answer is a design pattern. If the agent waits for the routing decision before speaking, the decision has to arrive within about 300 ms at the 95th percentile, and no hosted model reached that with acceptable accuracy. If the agent says “One moment, let me check who can best help you with that” while the decision runs, a second is enough, and Clef, Jev and Clef-flash all qualified.

Detours that taught me something

Billing. On Cloudflare’s free plan, my first request to Jev came back with “HTTP 402: Insufficient balance”. After I bought ten dollars of AI Gateway credit, Clef stopped working instead, because the free daily allocation had run out and Clef only uses the credit when a request is explicitly routed through a gateway. More than 10,000 decision-model calls cost less than three dollars in total, while the documentation gaps cost me several hours.

Self-hosting. llama.cpp serves the same API for Clef’s open weights, so I ran a 4-bit Clef-flash on my MacBook Pro (M4 Pro, 24 GB) while Teams and Outlook competed for memory. After raising the server’s batch size, which is too small by default for a typical request, the local model agreed with the hosted one on 96.5% of decisions. Its probabilities were absurdly flat, with 25% confidence on correct answers, until a single scaling factor fitted on development data made its calibration even better than the hosted version’s. At about two seconds per decision a laptop is no production server, but it proved that the open weights behave correctly outside Cloudflare.

An update in the middle of the benchmark. On October 9 Cloudflare announced faster serving, a cheaper Clef-flash and Clef-Omni, a model that also accepts audio. Clef’s median latency improved from 488 to 357 ms, but its 95th percentile stayed at about one second, roughly three times Cloudflare’s own figure.

Clef-Omni was the most interesting part of the project, because it receives the phone recording directly and skips speech recognition altogether. It was more accurate on the raw audio than on Whisper’s transcript, by two points on easy calls and eight on hard ones, and with text input it was the only model below 300 ms at the 95th percentile. Overall it still trailed Clef by almost seven points, although Cloudflare’s English benchmarks show them level. Given that it was three days old, had no tuned question variant and only heard synthetic voices, I read that gap as “not there yet”.

Decision models as judges. Our agent framework, Mastra, evaluates conversations with an LLM acting as a judge. On sixty synthetic dialogues about one guardrail, whether the agent revealed account details to an unverified caller, the LLM judge, Clef, Clef-flash and Jev all scored 60 out of 60, with the decision models two to four times faster at a fraction of the cost. The test was too easy to separate them on quality, yet a check that costs a fraction of a cent and answers in 400 ms can run on every conversation instead of a sample.

The moment I had to stop

Around day seven I had a latency Worker, a hard dataset, a review spreadsheet and a local inference setup, and I was planning a pipeline for anonymised real call recordings. Every step had been reasonable on its own, but together they were answering questions nobody needed answered to make a decision.

What we needed to know was whether the model is accurate enough, fast enough, cheap enough and honest enough about its uncertainty, and we had answers to all four. Everything else went onto a parked list, including the real-call idea.

The results

Macro-F1 against 95th-percentile latency on the 414 reviewed test utterances. The dashed lines mark the latency bars for blocking and non-blocking use.

“Auto-routable at 98% precision” is the share of calls a model can route without asking back while staying 98% precise. GPT 4.1 mini’s confidence was too coarse for that threshold.

What I would tell someone starting today

1. A well-prompted decision model routes German phone utterances about as accurately as a small LLM.
2. Category definitions move results more than model choice does.
3. A trustworthy confidence signal is worth more than a few points of accuracy.
4. Hosted decision models are four to eight times faster than LLMs but still too slow for an agent to wait on, so build non-blocking flows for now.
5. Audio-native routing already beats transcribe-then-classify for the same model.
6. Check the error column and have a human review your labels before you believe any number.

What comes next

We are not putting anything in front of callers based on synthetic data. The next step is a shadow mode in our voice simulator, where Clef classifies every turn alongside the existing agent without influencing the conversation, so that we can compare both decisions over many calls before routing becomes a non-blocking feature.

Three developments are on my watch list. Self-hosted Clef on proper GPUs is the only realistic path to blocking-grade latency with data that stays in our own infrastructure. The next version of Clef-Omni matters because skipping speech recognition is too attractive to ignore. And decision models as permanent guardrails, asking on every turn whether the caller is verified or wants a human, may turn out to be their most valuable use.

Two weeks ago I thought I was testing a faster classifier. I learned at least as much about my label definitions, my measurement mistakes and my tendency to over-engineer as about the models, and that is probably the most useful thing a benchmark can do.

---

All data in this benchmark was synthetic, and every test label was reviewed by a domain expert. Models and prices reflect the state of October 12, 2026.