Nov 9, 2026

Article

Medical Transcription API, How Clinical Speech-to-Text Works and How to Choose One

1 min read

A medical transcription API takes clinical audio and returns text as JSON, usually with timestamps, confidence scores and sometimes speaker labels. A medical speech to text API is the same product under another name, and neither one returns a clinical note. The thing to know before you buy one is that the most damaging errors look like correct output. In a study of 217 dictated clinical notes, speech recognition turned "The patient disagreed with this recommendation" into "The patient did agree with this recommendation", and "Inadequate evaluation to exclude neoplasia" into "Adequate evaluation to exclude neoplasia". Both are clean English. Neither raises an error.

Key Takeaways

A medical transcription API returns timed text, not a note. Turning that text into a note is a separate step with its own errors.
The failures that matter pass every technical check: a flipped negation, a wrong dose, or audio dropped before the connection was ready.
A small per-sentence fabrication rate is a large per-visit one. At 0.14% per sentence and 200 sentences a visit, about one visit in four contains invented text.
Word error rate scores a wrong dose and a wrong "so" the same. Measure recall on your own term list instead.
Run the same file twice and compare. It is simple, and it is how the Careless Whisper researchers detected fabricated text.

How Does a Medical Speech-to-Text API Work?

Take one invented sentence: "She denies chest pain, so increase metoprolol to fifty milligrams twice daily." It has twelve words. Assume it takes five seconds to say.

Diagram of a clinical sentence moving from microphone through recognition to JSON, then a note

1. Capture

A microphone samples the air pressure. At 16 kHz, 16-bit and mono, that is 32,000 bytes a second, so our sentence is 160,000 bytes.

16 kHz is the usual target. Google's speech guidance recommends 16,000 Hz or higher, and AWS's input docs treat 8,000 Hz as low-fidelity telephone audio. Sampling at 8 kHz can't hold anything above 4 kHz, and upsampling later can't restore it. That's why Google says to send 8 kHz audio as it is rather than resample it.

2. Encoding

Recorded and live audio travel differently. On Sully's streaming API, each message is JSON with base64 audio, and chunks of about 50 to 200 ms work well. A 100 ms chunk is 3,200 bytes, which base64 inflates to 4,268 characters. So our sentence arrives as 50 messages carrying about 213,400 characters of audio.

Recorded audio has a different constraint. Sully's upload API caps one file at 100 MiB and 24 hours, and both caps apply. Uncompressed 16 kHz mono reaches 100 MiB in 54.6 minutes (104,857,600 bytes ÷ 32,000 bytes a second). Fitting 24 hours into that file needs an average of about 9.7 kbps (104,857,600 × 8 ÷ 86,400 seconds), which is well into lossy compression, and Google warns that lossy codecs may reduce accuracy. You won't get lossless audio at both limits. The audio transcription API article covers uploads, and the real-time transcription API article covers streaming.

3. Recognition

Many vendors don't document their architecture. Whisper is a useful example because its paper is public. Audio is resampled to 16 kHz and converted to an 80-channel Mel spectrogram, using 25 ms windows with a 10 ms stride. That makes our sentence about 500 frames. The paper calls the decoder an "audio-conditional language model", and it also conditions on the transcript so far to resolve ambiguous audio.

So the system isn't looking words up. It is choosing likely text given the sound, which is why its mistakes read so well. "Fifty" can become "fifteen". AssemblyAI's example is "metoprolol" becoming "metronidazole", a beta blocker turned into an antibiotic. Vendors bias this step with medical modes and term lists. Sully's term list is word_boost, a streaming option.

4. Speaker labels

The output may attach a label to each word. On Sully's API it looks like "speaker": 0: a number, not a role.

Sully's note generation example passes text written as "Clinician: ... Patient: ...", so if you want that form, getting from numbers to roles is your code. If you can record each person on a separate channel, do it, because physical separation beats inference. Google recommends the same, and Sully's multichannel option returns one result per channel. The speaker diarization API article goes deeper.

5. The JSON

Here is our sentence in the shape Sully documents for a finished upload. This is illustrative. The field names come from the docs. The values are invented, not real API output, and only two words are shown.

{
  "data": {
    "id": "tx_...",
    "status": "completed",
    "result": {
      "channels": [
        {
          "transcript": "She denies chest pain, so increase metoprolol to fifty milligrams twice daily.",
          "confidence": 0.97,
          "words": [
            { "word": "denies", "start": 0.31, "end": 0.74, "confidence": 0.98, "speaker": 0 },
            { "word": "metoprolol", "start": 2.06, "end": 2.71, "confidence": 0.91, "speaker": 0 }
          ]
        }
      ]
    }
  }
}

{
  "data": {
    "id": "tx_...",
    "status": "completed",
    "result": {
      "channels": [
        {
          "transcript": "She denies chest pain, so increase metoprolol to fifty milligrams twice daily.",
          "confidence": 0.97,
          "words": [
            { "word": "denies", "start": 0.31, "end": 0.74, "confidence": 0.98, "speaker": 0 },
            { "word": "metoprolol", "start": 2.06, "end": 2.71, "confidence": 0.91, "speaker": 0 }
          ]
        }
      ]
    }
  }
}

{
  "data": {
    "id": "tx_...",
    "status": "completed",
    "result": {
      "channels": [
        {
          "transcript": "She denies chest pain, so increase metoprolol to fifty milligrams twice daily.",
          "confidence": 0.97,
          "words": [
            { "word": "denies", "start": 0.31, "end": 0.74, "confidence": 0.98, "speaker": 0 },
            { "word": "metoprolol", "start": 2.06, "end": 2.71, "confidence": 0.91, "speaker": 0 }
          ]
        }
      ]
    }
  }
}

Your code has to handle a few things:

words and speaker are optional. Don't read them unconditionally.
The upload response comes first. A 201 returns an ID and pending, with no text. You poll, or use webhooks if they are configured. The docs' SDK examples set maxRetries: 0 on creation, and a repeated POST can create duplicate billable work.
Streaming has a different shape. It returns text and is_final rather than channels and transcript, and timing can be null. If you use both paths, write an adapter early.

Test confidence on your own audio before you set a threshold on it. Nothing in the object says whether a word was actually spoken.

Where the Sentence Breaks at Each Stage

Stage	What goes wrong	What happens to our sentence	How to catch it
Capture	Distant mic, overlapping speech, automatic gain or noise reduction	"She denies" fades out. Google warns that overlapping speech may be treated as noise and ignored, and advises against gain control and noise reduction before sending.	Test real rooms. Set `autoGainControl` deliberately in browsers and compare results.
Encoding	Audio sent too early, or bytes that don't match the declared format	On Sully's streaming API, audio sent before `connected` is dropped and malformed messages are ignored. The opening words vanish without an error.	Hold audio until `connected`. Compare the first word's timestamp with the recording.
Recognition	Substitution, deletion or invented text	"Fifty" becomes "fifteen". "Denies" disappears. A sentence appears that nobody said.	Term recall on your own list, plus the double-run test below.
Speaker labels	Wrong or shifted labels	The denial is attributed to the wrong person. On a streaming API, a reconnect can also start a new session.	Score short turns separately. Don't assume `speaker: 0` is the same person after a reconnect.
Your code	Partial results stored as final	A provisional "fifteen" reaches the database.	Persist only `is_final` results.

Test noise too. A June 2026 preprint ran 272 OSCE-style clinical dialogues through Whisper large-v3 and a fixed LLM. Stationary ambient noise raised word error rate by just 0.71 percentage points and nearly doubled unsafe outputs, and the authors link the damage to errors on cues such as negation. It is one preprint, but it shows why a quiet-room demo tells you little.

Accuracy, briefly

In our twelve-word sentence, "fifty" becoming "fifteen" scores a word error rate of 8.3% (1 ÷ 12). So does "so" becoming "and". The metric can't tell them apart, which is the core of Sully's article on word error rate.

The dictation study from the introduction is still a useful map, though it covers a 2016 commercial system rather than a current one. Raw output had 7.4 errors per 100 words, falling to 0.4 after transcriptionist review and 0.3 in signed notes. Medication was the most common clinical error type in the raw output, and one error turned Rapamune into verapamil. Most of the safety came from humans reading the text.

Vendor accuracy numbers come from private test sets and can't be stacked against each other. Sully's 92.2% figure scores finished notes, not transcripts.

The error with no audio behind it: text nobody said

Koenecke and colleagues ran 13,140 sentence-length audio segments through OpenAI's Whisper API in April and May 2023. The body of the paper reports that 1.4% produced hallucinated text (the abstract rounds this to "roughly 1%"), and 38% of those hallucinations contained explicit harms. Their harm categories include invented health statuses. In one example, the speaker said "pick the bread and peanut butter", and the output added "In a large mixing bowl, combine the softened butter."

The per-sentence rate undersells this. Assume a visit has 200 sentence-length utterances:

At 1.4%, you'd expect 2.8 fabrications a visit, and 94% of visits would have at least one (1 − 0.986²⁰⁰).
At a tenth of that rate, 0.14%, 24% of visits would have at least one (1 − 0.9986²⁰⁰).

This is arithmetic, not a forecast. It assumes errors are independent, the audio came from AphasiaBank research interviews rather than clinic visits, and the model has changed: in a December 2023 retest, 12 of the 187 problem segments still fabricated. But a rate that sounds negligible per sentence isn't negligible per chart.

Three cautions:

It doesn't prove other vendors are safe. The authors found no comparable fabrications from Google, Amazon, Microsoft, AssemblyAI or Rev on those 187 segments, and called it OpenAI-specific at the time. But those segments were chosen because Whisper failed on them.
The mechanism isn't settled. OpenAI's model card hypothesizes that the model mixes next-word prediction with transcription. Corti argues that attention-based encoder-decoder models are especially prone to it, though Corti sells a competing API.
Current rates are unknown. A search for an independent, multi-vendor fabrication study on clinical audio turned up nothing recent. Treat every vendor as untested until you test it.

You can borrow the researchers' method. Hallucinations are often non-deterministic, so they compared two runs of each segment for multi-word differences where the output ran longer than the speech. Do the same on your own audio, and include long pauses, because longer non-vocal stretches predicted hallucination in the study. As an extra filter, which is this article's suggestion rather than the paper's, flag words whose timestamps fall inside stretches a voice activity detector marks as non-speech. A fabrication that repeats on both runs will slip past a diff. The Whisper alternatives article covers the field reports and the shutdown.

A Transcript Is Not a Clinical Note

A transcript tries to record what was said. A note decides what matters. To reach "Denies chest pain. Increase metoprolol to 50 mg BID", something has to resolve "she", file the denial under history, file the change under plan, and convert spoken words into a dose. Each of those steps can introduce errors of its own.

A note model faithfully summarizing a wrong transcript produces a confident, well-formatted wrong note. Keep the word timestamps, so a reviewer can jump from any dose to the audio behind it.

Asgari and colleagues had clinicians review 450 LLM-drafted notes from transcripts. They found hallucinations in 1.47% of note sentences, and 44% of those were major. 30% were negations, the same class of error that speech recognition produces. 3.45% of transcript sentences were omitted. The same paper reports that refining prompts and workflows brought major errors below previously reported human note-taking rates. Note generation can work well, but only if someone measures it.

On Sully's API, Note Generation takes a transcript and your template and returns the note as a rendered string. Text to JSON takes text and your JSON Schema and returns validated data. They are separate requests, and on billing-enabled accounts they are separately billable.

Their limits don't match the upload limit. Note Generation takes up to 100,000 characters of transcript and Text to JSON up to 50,000. At an assumed 120 words a minute and 5.5 characters a word (660 characters a minute), those limits are about 150 and 75 minutes of talk. A full 24-hour upload would be about 950,000 characters. Long recordings need a plan before they reach the note step.

The ambient scribes article covers what a scribe needs from the speech layer, and whether to buy parts or one API.

Buying Checklist for a Medical Transcription API

Name your last artifact. Is it a transcript, a formatted note or validated fields? The gap between that and what the vendor returns is what you will build.
Choose recorded, live or both. Then read the upload and streaming limits, including what a reconnect does.
Separate dictation from ambient. One clinician at a microphone is a different problem from a conversation. See the medical dictation API article.
Test 20 to 50 of your own recordings. Score recall on your own drug and dose list, and count negation flips by hand. Score them with the word error rate method.
Run the double-run test, with long silences left in.
Score speaker labels on short turns, where "no" and "yes" live, rather than on long monologues.
Test language switching, if your patients do it. A long language list doesn't answer this, and the multilingual article covers why.
Read the contract, not the badge. Check who signs the BAA, for which entity type, and what is retained by default. Start with whether a medical transcription API is HIPAA compliant.
Price what you will ship, with add-ons switched on. See speech-to-text API pricing.

For names, use the medical API comparison or the general speech-to-text comparison. If you already use a specific vendor, there are separate articles on Deepgram, Whisper and Amazon Transcribe Medical.

Where Sully Fits

Sully wrote this article, so weigh this section accordingly.

Sully covers the whole path above on one API: uploads, streaming, note generation and Text to JSON. The API page lists 100k+ notes a month, 40+ specialties, 99.96% uptime, and "Tebra's AI Scribe built on Sully". It also lists ISO 27001, HIPAA, HITRUST, SOC 2 Type II and Zero Data Retention, and it says Sully doesn't use or train on customer data. Sully deletes the audio, transcript, and input and output tokens after processing, and the transcription API also has a DELETE endpoint, so you can remove a transcript once you've saved it.

Another vendor can be the better choice. If you upload recordings after the visit and need term biasing on that path, AssemblyAI says it takes up to 1,000 keyterms on async audio.

The trust page says Sully "operates as a HIPAA Business Associate, executing BAAs with every covered entity". It also signs BAAs with software companies building on the API.

To test it, get a key from the Sully API page and start with the transcription docs. Run 20 of your own recordings twice, and check every dose against the audio.

FAQ

What is a medical transcription API?

It's a web service that takes clinical audio, as a file or a live stream, and returns text as structured data, often with word timestamps and confidence scores. It doesn't produce a chart-ready note on its own.

What is the difference between a medical and a general speech-to-text API?

Mostly vocabulary tuning and how it's sold. AssemblyAI adds a domain parameter to its base model, and ElevenLabs offers a separate Scribe v2 Medical model. The improvements vendors report are their own measurements, so test them on your audio.

Can a medical transcription API write the clinical note?

Not by itself. A note needs a separate generation step, whether from the same vendor or another, and that step has its own hallucination and omission risks. Sully offers it as a separate Note Generation request.

Do speech-to-text APIs make things up?

They can. Peer-reviewed work found Whisper inventing whole phrases in 2023, and OpenAI's model card acknowledges the behavior. Rates for current models on clinical audio aren't well established, so test your vendor with repeated runs and long silences.

Is a medical transcription API HIPAA compliant?

An API isn't compliant by itself. It depends on the BAA, how the vendor handles data, and how you build on it. The HIPAA article separates those questions.