Introduction

One of the most common greetings that Arabic-speaking front desk receptionists hear is the courtesy phrase «يعطيك العافية». On calls where English and Arabic may be mixed, an English speech recognizer could incorrectly transcribe it phonetically as "Mary, I'll take it off here," and report 98% confidence. If a voice agent trusts this, it could give an incorrect reply in the wrong language.

Language selection is one of several decisions that voice agents must make in a fraction of a second during a call. Has the caller finished speaking? Was an "okay" a confirmation? Does the request belong to another desk? A wrong decision can cause the agent to misunderstand a request, route to the wrong destination, or take incorrect actions.

We use predefined rules to resolve most of these cases, but some decisions are more nuanced than the rules can account for. We now use decision models for these. Decision models, sometimes called “System One” models, answer questions by choosing from a set of possible answers. Given a description, a question, and the available options, the model returns a probability for each option. Because it does not generate text, it can respond quickly enough to use during a live call.

In this article, we describe how we use one decision model, mammoth-1 (deployed by Baseten in the UAE), to choose between Arabic and English for hospital calls in the Middle East region. We report results from one week of calls.

• Corrections: In its first three days, the decision model disagreed with our predefined rules on 19% of its decisions. When taking a probability of 0.8 as a threshold, the decision model was correct 93% of the time.

• Response time: The median response time was 471 ms, with a p95 of 675 ms.

Why we chose decision models

In the Middle East region, our receptionist agent greets callers in Arabic, then handles appointments, identity checks, and hospital questions in whichever language they use. However, more than half of callers use Arabic and English, sometimes within the same sentence. We needed a speech-to-text setup that could capture both reliably.

We tried an Arabic-only recognizer, but it transcribed English phonetically in Arabic script and produced mistranslations. A bilingual recognizer also struggled, omitting short Arabic replies and misrecognizing English names. Of the setups we tested, two recognizers running in parallel—one configured for Arabic and one for English—performed best.

However, this left a second problem: deciding which transcript to use. We first compared the recognizers’ confidence scores, but each score assumes that the caller spoke the recognizer’s expected language. On one call, the Arabic recognizer returned “I have very bad pain in my knee” with 0.96 confidence, while the English recognizer returned “And the” with 0.95. In other cases, an incorrect transcript received the higher score.

Mixed-language speech illustrates why choosing a transcript could be difficult. For example, one caller asked in Arabic, «ازاي اقدر اطلب الميديكال ريبورت بتاعي» (“How can I request my medical report?”), using the English term “medical report” within the sentence. The English recognizer returned only “Medical report.” It picked up the English term but lost the surrounding request. Selecting that transcript would leave the agent without the caller’s actual question.

Detecting the language directly from the audio was also unreliable on short phrases. A small Whisper model assigned over 90% probability to English for an Arabic courtesy phrase. Clips under two seconds from one Arabic call were classified as Swedish, Hebrew, German, French, and Turkish.

We built an arbiter pipeline that first applies predefined rules to choose between the two transcripts. These rules recognize common phrases and weigh recognizer confidence alongside a preference for the current conversation language. When the rules cannot settle the choice, the pipeline passes both transcripts to the decision model.

Using a decision model for ambiguous turns

Decision models return a choice and probabilities for predefined options, given an input and a typed question. TypeSafe AI introduced the format with Jev in September 2026 [1], followed by Baseten's release of mammoth-1 and elephant-1 in October [2, 3].

The decision model chooses between the two transcripts using conversation context. It receives the agent's last utterance, the caller's recent utterances with their languages, and both candidate transcripts. Its task is to select the recognizer that captured what the caller said. From this decision, the receptionist agent then uses the returned probability to decide whether the agent should continue in the selected language or ask for clarification.

We chose to test mammoth-1 and elephant-1. Both models have open weights, which allowed us to deploy them in the UAE to meet data residency and privacy requirements.

Comparing mammoth-1 and elephant-1

We first tested mammoth-1 (27 billion parameters) and elephant-1 (9 billion parameters) on 106 caller turns from 36 hours of real calls. We reviewed turns where the models chose different transcripts and manually scored those with a clear correct answer. On these labeled disagreements, mammoth-1 was correct 77% of the time, compared with 54% for elephant-1.

The largest difference was on short replies, where conversation context may be the only way to interpret "yes," "no," or "okay." Among the labeled disagreements involving short replies, mammoth-1 was correct in every case, compared with 50% for elephant-1. This matters because most turns sent to the model contain three words or fewer. elephant-1 also tended to classify English speech transcribed in Arabic script as Arabic, whereas mammoth-1 correctly recognized it as English.

Results in production

After selecting mammoth-1, we deployed it on ambiguous turns in live calls. We wanted to understand whether its choices improved on the rules’ selections and whether it could respond quickly enough for a live conversation.

The evaluation below covers mammoth-1’s 545 decisions during its first three days in production. We examined whether it corrected mistakes made by the rules, how its reported probabilities could guide acceptance thresholds, and how quickly it responded.

Tuning when to trust the decision model

The decision model's disagreements with the rules let us examine whether it corrected errors made by the rules or introduced new ones. mammoth-1 disagreed with the rules on 104 of its 545 decisions (19%). We reviewed those disagreements to determine whether it corrected an existing error or introduced a new one.

In initial testing, we used a probability threshold of 0.6. Above this threshold, we would select the transcript that was proposed. If no decisions exceeded this threshold, the voice agent would ask the caller to just repeat their utterance. Reviewing results showed that this cutoff was too permissive. Only 31% of reviewed overrides between 0.6 and 0.8 corrected an error, compared with 92% between 0.8 and 0.9 and 93% at 0.9 or above.

These results led us to propose requiring a probability of at least 0.8 before accepting a selection that disagreed with the rules. Applied to the reviewed disagreements, this policy would have prevented 73% of mammoth-1’s mistakes while preserving 91% of its corrections.

Raising the threshold may reduce errors while prompting more callers to repeat themselves. In 2% of cases, both candidate transcripts conveyed the same meaning—for example, “No. No.” and «لا لا». Either transcript would have been acceptable, but the threshold probability triggered a request to repeat.

Overall, across all 545 answers, the median reported probability of the decisions was 0.93, and 75% of cases scored 0.8 or above. These probabilities let us assess threshold changes before deploying them. Using labeled decisions, we can count how many correct answers a higher threshold would reject and how many errors it would prevent.

Speedup with decision models

Response time is critical in voice agents. Every additional decision increases the pause between the caller finishing and the agent replying. Slow responses make conversation harder to sustain and increase the probability of an unsuccessful outcome. Decision models avoid the delay of generating a text response by scoring the available options in a single pass over the conversation [2]. In our calls, the median response took 471 ms, with 95% arriving in under 675 ms and 99% in under 766 ms.

Conclusions

We use two parts together: rules handle straightforward turns, while the decision model uses conversation context when the choice is less clear. We expected the second step—the decision model—to be occasional, but the judge step was needed on 36% of turns in which both recognizers returned a transcript. Ambiguous decisions were more common than we expected, which suggests decision models may help with other choices a voice agent makes.

The other lesson was that choosing an answer and accepting it are separate decisions. The model could correct mistakes, but it could also introduce them. Reviewing its choices helped us set a stricter acceptance threshold while understanding the trade-off: fewer errors, but potentially more requests for callers to repeat themselves.

We started with language selection because it was the hardest decision in our current voice agent deployments. The same approach may help with end-of-turn detection, consent, and routing. Each task can require the agent to resolve ambiguity in a fraction of a second.

References

1. TypeSafe AI. "Introducing System One Models & Jev." September 15, 2026. https://typesafe.ai/blog/introducing-system-one-models-and-jev

2. Base Labs, Baseten. "mammoth-1 and elephant-1: Frontier Decision Models." October 21, 2026. https://labs.baseten.co/articles/mammoth-1-and-elephant-1-frontier-decision-models

3. Baseten. "Decision models." Model API documentation. https://docs.baseten.co/inference/model-apis/decision-models

4. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. "Robust Speech Recognition via Large-Scale Weak Supervision." Proceedings of the 40th International Conference on Machine Learning, 2023.