A transcription model can post an excellent word error rate and still be the wrong model for a clinical product. The words it gets wrong are the ones that carry the risk: the medication names, the dosages, the "no" sitting in front of a diagnosis.

That is not a failing of any particular vendor. It is a property of the metric everyone reports.

Key Takeaway: Word error rate is the share of words a speech recognition system gets wrong, calculated as substitutions plus insertions plus deletions, divided by the total number of words in a reference transcript. On medical audio it is misleading on its own, because it weights every word equally while clinical risk concentrates in a handful of rare ones. Published benchmark bands put medical dictation at 2 to 8 percent, but peer-reviewed studies of real clinical audio report a 7.4 percent error rate on dictated notes, a 25 percent word error rate on conversational clinical audio, and 34 percent on the sentences clinicians flagged as harm-related. Medication names are the most common clinical error type in speech recognition output. Score a transcription vendor on your own medication and specialty term list rather than on aggregate word error rate. Clinical transcription APIs built for this workload, Sully among them, let you boost those terms in the API call rather than correcting them afterwards.

What Word Error Rate Measures in a Clinical Transcript

The Formula and What Each Error Type Means

Word error rate is the sum of substitutions, insertions and deletions, divided by the number of words in the reference transcript [1].

Three error types, and the difference between them matters in a clinical context:

• Substitution. The system heard a word and wrote a different one. This is where a drug becomes another drug.

• Insertion. A word appears in the transcript that nobody said.

• Deletion. A spoken word is missing from the transcript.

Lining the two transcripts up uses Levenshtein edit distance, which picks whichever alignment produces the lowest score [2].

One quirk worth knowing before you read any vendor's chart. Insertions have no ceiling, so word error rate can exceed 100% on a short passage [2]. A system that adds six words to a four-word sentence scores 150%.

What Counts as a Good Word Error Rate

These are the bands you will see quoted [3]:

Broken out by domain, the same source puts voice assistants at 5 to 10%, broadcast transcription at 8 to 15%, telephony at 15 to 25%, and medical dictation at 2 to 8% [3]. Human transcription runs 2 to 4% under good conditions [3].

Hold onto that medical number. The next section tests it against clinical data.

Why the Metric Treats a Drug Name Like a Filler Word

Every word in the denominator counts once. "The" and "hydrochlorothiazide" are worth exactly the same.

So a model can miss most of the clinically loaded terms in a transcript and still post a strong aggregate score, because those terms are a small share of the total words.

Speechmatics makes the point with a non-medical example. Drop "suspected" from in front of "murderer" and you have a 20% word error rate and a sentence that is defamatory [2]. Apple's research group makes the same point by building an alternative that counts only errors which change meaning, hurt readability, or misspell proper nouns [4].

Both are right. Neither opens a clinical dataset.

What Happened When Researchers Measured It on Real Clinical Audio

The published bands describe something real. The question is whether it is the audio your product records.

Three peer-reviewed studies have measured speech recognition on actual clinical audio. None of them landed inside the 2 to 8% band.

Read each one with its denominator attached. The three studies measured different things in different settings, and flattening them into a single trend line would repeat exactly the mistake this article is about.

Dictated Notes Came In Above the Published Band

Zhou and colleagues, JAMA Network Open, 2018. They analyzed 217 notes dictated by 144 physicians across two institutions, covering office notes, discharge summaries and operative notes [5].

The error rate in the raw speech recognition output was 7.4%, meaning 7.4 errors per 100 words. It fell to 0.4% after a professional transcriptionist reviewed it, and 0.3% in the note the physician finally signed [5].

Two things to pull out of that.

First, 7.4% sits at the top of the published 2 to 8% band rather than in the middle. And this was dictation, which is the easiest setting speech recognition gets in a clinical environment: one speaker, deliberate pace, usually a quiet room.

Second, and this is the finding the article is named after: medication was the most common clinical semantic type of error in the raw speech recognition output [5].

One number from that study needs handling with care. Of the errors found in the speech recognition output, 5.7% were clinically significant [5]. That is a share of errors. Not a share of notes, and not a share of words.

Emergency Department Notes Were Worse

Goss, Zhou and Weiner, International Journal of Medical Informatics, 2016, summarized by the US Agency for Healthcare Research and Quality's Patient Safety Network [6].

71% of notes dictated by emergency medicine physicians contained errors, and 15% of those errors were considered critical [6].

Different denominator again, and the difference is the point. This is the share of notes containing at least one error, not an error rate per word. A system can post a single-digit word error rate and still put a mistake in seven notes out of ten, because errors cluster rather than spreading evenly. A product team needs both framings, and conflating them is how an evaluation goes wrong.

Conversational Audio Was Worst Where It Mattered Most

Miner and colleagues, npj Digital Medicine, 2020. They ran 100 psychotherapy sessions through a HIPAA-compliant commercial cloud speech recognition service [7].

The overall transcription word error rate was 25%.

For the sentences a clinician had independently identified as harm-related, it was 34% [7].

Sit with that pair for a second. The errors did not average out across the transcript. They concentrated in the content that carried the risk.

The same paper notes that prior literature puts conversational medical speech recognition somewhere between 18 and 63% [7].

The caveats are real and worth stating. This is one study, in one specialty, on one commercial system, published in 2020, and speech models have improved a great deal since. What has not changed is the shape of the result. Work presented at Interspeech 2024 found the same pattern on accented clinical speech: overall word error rates can look acceptable while errors on clinical entities run higher [8].

Why General Speech APIs Break on Medical Vocabulary

Drug Names Are Rare Tokens

A general speech model is optimized for average word accuracy across a training distribution dominated by ordinary speech.

Medication names are rare in that distribution. The model has seen far fewer examples of them, and it has weaker expectations about what comes before and after.

The model is doing exactly what it was built to do. The mismatch is that a clinical product cares most about the part of the distribution the model saw least.

Look Alike and Sound Alike Names Defeat Context

Language models resolve ambiguity using context, which works when context narrows the options.

Drug names often defeat that. The confusable alternative is usually also a drug, also plausible in a clinical sentence, and sometimes also a reasonable dose. Context cannot choose between two candidates that both fit.

This is why term boosting is sold as its own feature rather than running by default, and why it comes with a ceiling. Deepgram's keyterm prompting handles up to 100 terms per request [9]. For a product covering several specialties, 100 is a budget you have to spend carefully.

What Clinical Speech Recognition Has to Get Right That General Speech Does Not

Six categories, and they double as the start of your term list:

• Medication names, generic and brand

• Dosages and units

• Negation: no, denies, ruled out, no evidence of

• Anatomy and laterality: left, right, bilateral

• Specialty vocabulary

• Clinician and patient names

Negation is the quiet one. It is ordinary English, so it does not look like specialty vocabulary and no term list built from a formulary will catch it. Dropping a single "no" inverts the clinical meaning of a sentence while moving word error rate by exactly one word.

How to Measure Transcription Accuracy on Clinical Audio

1. Build the Term List Before You Build the Test Set

Start from your own formulary and specialty vocabulary rather than a public word list. Your top 200 medications with their generic and brand variants, the dosage formats you actually see, your negation phrases, and the clinician names that recur in your audio.

The list defines what you are measuring. Building it first is what stops you from reverse-engineering a flattering test once you have seen the results.

2. Score Term Recall Separately From Word Error Rate

Report two numbers, not one.

Aggregate word error rate tells you whether the transcript reads correctly. Recall on your term list tells you whether it is safe to act on. A candidate that wins the first and loses the second is the wrong model for a clinical product, and a single figure cannot show you that.

This is the direction the research is taking. The Interspeech 2024 work reports medical named-entity recall alongside word error rate [8], and a September 2026 preprint proposes MedWER, scored against a fixed list of 19,373 clinical terms [10]. Neither is a standard yet, so the scoring is yours to build. Weighting the terms during transcription is not: word boosting is a parameter on several clinical speech APIs today, including Sully's.

3. Weight Errors by Clinical Consequence, Not by Count

Not every miss costs the same, so stop counting them as though they do.

Sort your errors into ones that change clinical meaning (wrong drug, wrong dose, dropped negation, wrong side) and ones that do not (filler words, formatting, punctuation). Report the consequence-weighted count next to the raw one.

Apple's group did this for readability and found 9.2% word error rate against 1.4% once only the meaningful errors counted [4]. That gap is how much of a raw score can be noise.

4. Test Conversation, Not Just Dictation

The published bands describe dictation: one speaker, deliberate pace, usually a quiet room. Almost nothing an ambient clinical product records looks like that.

The measured gap between the two settings is the difference between 7.4% and 25% in the studies above [5][7]. So a vendor evaluation run on clean dictated samples tells you very little about how an ambient build will behave.

Record the real thing, with crosstalk, interruptions, and whatever room you actually work in.

5. Rerun the Benchmark on Every Model Version

Speech models get updated under the same endpoint, so the number you qualified a vendor on has an expiry date nobody tells you about.

Keep the test set and the term list in version control next to your code and run them on each release. That also gives you the only longitudinal accuracy data anyone in your category will have, because no vendor publishes theirs over time.

How Sully Approaches Accuracy on Clinical Audio

You have just been handed five tests. Here is how Sully answers them, including the one where we have no number to hand you.

Sully is a research company. It builds and runs its own clinical models, then sells access to them as APIs [11].

Word Boosting, Notes and Structured Fields in One API

Word boosting.Streaming transcription lets you boost the words that matter, such as drug names [11]. That is checks one and two answered inside the transcription call, with your term list as a parameter rather than a post-processing step.

Conversational audio. Transcription handles recorded visits up to 24 hours long across 30+ languages, batch or streaming, with polling or a webhook when the transcript is ready [11]. Check four can be run on real encounters rather than on dictation samples.

Where errors become visible. Note generation takes a transcript and your own note template and returns the finished note. Text to JSON takes your JSON Schema and returns validated fields, ready for an EHR [11]. This is where a dropped negation stops being a word buried in a paragraph nobody rereads and becomes a wrong field you can catch.

Clinical reasoning. Sully M1 is a clinical LLM called like any chat model, in OpenAI request format, for the work the task APIs do not cover [11]. No second model vendor, no second BAA.

Data handling. Zero Data Retention is the published default, nothing stored after processing, and Sully does not train on customer data. ISO 27001, HIPAA, HITRUST and SOC 2 Type II, with a BAA signed with every customer [11][12]. Worth checking before you pipe real encounter audio through anyone's benchmark.

Someone has already shipped a product on this. Tebra's AI Scribe is built on Sully, which is the proof point that belongs to the API rather than to the platform.

Across all customers, the APIs handle more than 100,000 notes a month across 40+ specialties at 99.96% uptime and an 11.23 second p50 [11].

What Sully Does and Does Not Publish

In an article about word error rate, this is worth saying plainly rather than leaving for you to notice: Sully publishes no transcription word error rate.

What Sully does publish is 92.2% clinical note accuracy [11]. Be precise about what that measures. It scores the finished note, not the transcript underneath it. Those are different artifacts, measured different ways, so setting our note figure beside somebody's transcription figure would be the exact comparison this article just spent a section arguing against.

Which leaves you where the research leaves everyone. The only accuracy number that can decide a clinical build is the one you generate yourself, on your own recordings, against your own drug list.

Run it. Get an API key at dashboard.api.sully.ai, point it at 20 of your own encounters, and count the medications. Change the base URL and the model name. There is no demo to sit through.

FAQ

How do you calculate word error rate?

Word error rate is the sum of substitutions, insertions and deletions, divided by the total number of words in the reference transcript, usually expressed as a percentage [1]. A substitution is a word transcribed incorrectly, an insertion is a word that appears in the transcript but was never spoken, and a deletion is a spoken word that is missing. The alignment between the two transcripts is found using Levenshtein edit distance, which picks the alignment producing the lowest score [2]. Because insertions are counted without a ceiling, word error rate can exceed 100% on short passages [2].

What is a good word error rate for medical transcription?

Commonly published bands treat 0 to 5% as excellent and 5 to 15% as good, with medical dictation listed at 2 to 8% [3]. Peer-reviewed measurements on real clinical audio sit above that: 7.4% on dictated clinical notes [5], and 25% on conversational clinical audio, rising to 34% on sentences clinicians flagged as harm-related [7]. The honest answer is that a single target number is the wrong goal. A transcript at 10% word error rate that captures every medication is more useful to a clinical product than one at 5% that does not.

Why do speech recognition systems get drug names wrong?

Medication names are rare tokens in the general speech data these models are trained on, so the model has fewer examples of them and weaker expectations about the words around them. Look-alike and sound-alike drug names make it worse, because the confusable alternative is usually also a drug and also plausible in the sentence, so context cannot resolve it. Zhou and colleagues found medication to be the most common clinical error category in raw speech recognition output across 217 dictated clinical notes [5]. Term boosting features help, though they carry caps: Deepgram's keyterm prompting handles up to 100 terms per request [9].

Is word error rate a reliable way to compare transcription vendors?

Not on its own, for two reasons. The metric weights every word equally, so it cannot tell a dropped filler word from a dropped negation that inverts a diagnosis [2][4]. And vendor-published figures are each measured on private test sets against unnamed comparison systems, so two numbers from two companies describe two different tasks and cannot be stacked against each other. Use published figures to build a shortlist, then settle the decision on your own recordings scored against your own term list.

What should you measure instead of word error rate for clinical audio?

Measure three things together: aggregate word error rate for readability, recall on your own medication and specialty term list for clinical safety, and an error count weighted by clinical consequence. Research is converging on this. Interspeech 2024 work reports medical named-entity recall alongside word error rate [8], Apple's group reports a readability-weighted variant [4], and a 2026 preprint proposes scoring against a fixed clinical term list [10]. Sully's streaming transcription lets you boost the terms on that list directly in the API call, and its note generation and Text to JSON endpoints return the finished note and schema-validated fields, where a clinical term error shows up as a wrong field rather than a word buried in a paragraph [11].

Sources

[1] Rev — What Is WER? What Does Word Error Rate Mean? [2] Speechmatics— The Problem with Word Error Rate (WER) [3] LlamaIndex — Understanding Word Error Rate for Speech Recognition Systems [4] Apple Machine Learning Research — Humanizing Word Error Rate for ASR Transcript Readability and Accessibility [5] Zhou L, Blackley SV, Kowalski L, et al., JAMA Network Open— Analysis of Errors in Dictated Clinical Documents Assisted by Speech Recognition Software and Professional Transcriptionists[6] Agency for Healthcare Research and Quality, Patient Safety Network— Incidence of speech recognition errors in the emergency department [7] Miner AS, Haque A, Fries JA, et al., npj Digital Medicine — Assessing the accuracy of automatic speech recognition for psychotherapy[8] Afonja T, Olatunji T, Ogun S, et al., Interspeech 2024 — Performant ASR Models for Medical Entities in Accented Speech [9] Deepgram — Medical Transcription Speech to Text API [10] Behling J, arXiv preprint — MedWER: A Reproducible, Model-Free Evaluation Protocol for Medical Speech Recognition [11] Sully.ai — Sully API [12] Sully.ai— Trust Center