Generating a clinical note from a transcript is the easy part of building an AI scribe. Knowing whether the note is any good is harder, because you need a conversation and the note a clinician would have written from it. Real pairs like that are protected health information.

So we gathered 536 role-played doctor-patient conversation transcripts from four openly licensed research datasets, and 264 of them come with the clinician's own note. There are also 1,701 short exchanges, each paired with the one note section it should produce. It's all free on Hugging Face and in the Clinical Scribe Kit on GitHub. Sully maintains it and sells a clinical notes API, so the examples use Sully, but the data works with any model.

Key Takeaways

• 536 full visits, about 680,000 words. US outpatient, UK primary care and Canadian history taking, all role-played, so there's no patient data.

• 264 visits have an answer key. The 207 ACI-Bench and 57 PriMock57 visits include the note the clinician wrote, which you can compare your generated note against.

• Test whole notes and single sections. ACI-Bench checks the full note. MTS-Dialog's 1,701 snippets check one section at a time, such as medications or allergies.

• Word-overlap scores aren't enough. ROUGE rewards copying the reference's wording. Check for missing facts and made-up facts, and have a clinician read a sample.

What's in the Transcript Set

Every transcript uses the same plain-text format, one turn per line, labeled Clinician:, Patient: or Family member:. That's the format a notes API expects, so you can send it without any cleanup.

A visit and its note

Here's the start of ACI-Bench visit D2N033, a right knee injury:

And part of the note the clinician wrote from it:

That's the answer key. Your generated note should reach the same plan, keep the patient's history of diabetes, and not invent an exam finding that nobody said out loud.

Three Ways to Test Note Generation

Whole notes with ACI-Bench

ACI-Bench is the main set for transcript-to-note tests. Its notes follow one structure: chief complaint, history of present illness, review of systems, physical exam, results, and assessment and plan. The 207 visits come in three styles, and they stress a scribe in different ways:

The dictated visits are a useful test on their own. The clinician restates parts of the visit in a dictated summary after the conversation. Your note has to use both without counting anything twice. The medical dictation API guide covers that workflow.

Single sections with MTS-Dialog

Each of the 1,701 MTS-Dialog items is a short exchange plus the one section it should become. It's the quickest way to find out which sections your prompts or templates get wrong.

The rest cover assessment, physical exam, diagnosis, plan and a handful of smaller sections. Medications and allergies are short, but they're where a wrong word does the most harm, so score them separately.

Audio to note with PriMock57

PriMock57 is the only part with both audio and a clinician's note, so it tests the full chain. Transcribe the recording with your speech API, generate a note from your transcript, then generate a second note from the human transcript. If the two notes differ, the speech step caused it. The companion guide to the audio covers the transcription half.

Generate Notes From the Dataset

The encounters config on Hugging Face loads in one line. Each row has the transcript, the reference_note when the source has one, the source, the license and the chief_complaint.

import os, requests
from datasets import load_dataset

visits = load_dataset(
    "SullyHQ/open-clinical-conversations", "encounters", split="train"
)
aci = visits.filter(lambda v: v["source"] == "ACI-Bench")

# One of the 19 templates in the Clinical Scribe Kit
template = requests.get(
    "https://raw.githubusercontent.com/aniquethaplawala/"
    "clinical-scribe-kit/main/templates/outpatient-soap.json"
).json()

headers = {
    "X-Account-Id": os.environ["SULLY_ACCOUNT_ID"],
    "X-Api-Key": os.environ["SULLY_API_KEY"],
}

for visit in aci.select(range(5)):
    r = requests.post(
        "https://api.sully.ai/v3/notes",
        headers=headers,
        json={"transcript": visit["transcript"], "template": template},
        timeout=300,
    )
    r.raise_for_status()
    print(visit["encounter_id"], visit["chief_complaint"])
    print(r.json()["note"][:400], "\n")

import os, requests
from datasets import load_dataset

visits = load_dataset(
    "SullyHQ/open-clinical-conversations", "encounters", split="train"
)
aci = visits.filter(lambda v: v["source"] == "ACI-Bench")

# One of the 19 templates in the Clinical Scribe Kit
template = requests.get(
    "https://raw.githubusercontent.com/aniquethaplawala/"
    "clinical-scribe-kit/main/templates/outpatient-soap.json"
).json()

headers = {
    "X-Account-Id": os.environ["SULLY_ACCOUNT_ID"],
    "X-Api-Key": os.environ["SULLY_API_KEY"],
}

for visit in aci.select(range(5)):
    r = requests.post(
        "https://api.sully.ai/v3/notes",
        headers=headers,
        json={"transcript": visit["transcript"], "template": template},
        timeout=300,
    )
    r.raise_for_status()
    print(visit["encounter_id"], visit["chief_complaint"])
    print(r.json()["note"][:400], "\n")

import os, requests
from datasets import load_dataset

visits = load_dataset(
    "SullyHQ/open-clinical-conversations", "encounters", split="train"
)
aci = visits.filter(lambda v: v["source"] == "ACI-Bench")

# One of the 19 templates in the Clinical Scribe Kit
template = requests.get(
    "https://raw.githubusercontent.com/aniquethaplawala/"
    "clinical-scribe-kit/main/templates/outpatient-soap.json"
).json()

headers = {
    "X-Account-Id": os.environ["SULLY_ACCOUNT_ID"],
    "X-Api-Key": os.environ["SULLY_API_KEY"],
}

for visit in aci.select(range(5)):
    r = requests.post(
        "https://api.sully.ai/v3/notes",
        headers=headers,
        json={"transcript": visit["transcript"], "template": template},
        timeout=300,
    )
    r.raise_for_status()
    print(visit["encounter_id"], visit["chief_complaint"])
    print(r.json()["note"][:400], "\n")

Note Generation is synchronous, so the note comes back in the response. Every request carries a template with overall instructions and a list of sections. The kit has 19 of them: SOAP, history and physical, progress note and 16 specialties. If you want to see what each section should hold, the SOAP note template guide and the progress note template walk through them.

How to Score a Generated Note

The research papers behind ACI-Bench and MTS-Dialog report ROUGE and BERTScore, and those are fine for tracking whether a change made things better or worse. They measure how much your wording overlaps with the reference. A note can reuse the reference's words and still say the wrong knee.

For decisions that matter, score two things by hand or with a careful LLM judge, and spot-check the judge against a clinician:

• Missing facts. List the facts in the reference note, such as each problem, medication, dose and plan step. Count how many your note leaves out.

• Unsupported facts. List the facts in your note and check each one against the transcript, not the reference. Anything the transcript doesn't support is made up, even if it sounds clinically reasonable.

Then watch where facts land. A symptom the patient denied belongs in review of systems as a negative, not in the history of present illness as a complaint. Section-level tests on MTS-Dialog catch most of those.

Hard Cases in the Data

Role-played visits are tidier than real ones, but the set still has cases that break naive pipelines:

• People correcting themselves. "Maybe 8 days ago" after "a week back". The note should keep the corrected version.

• Family members speaking. Five visits include a Family member: speaker. What a relative reports isn't the same as what the patient says, and a good note says who said it.

• Non-US English. UK patients take paracetamol and see a GP. Your template decides whether the note keeps that wording.

• Dictation after the visit. The 40 dictated ACI-Bench visits mix a conversation with a clinician's summary, which tests whether the model double counts.

What These Transcripts Won't Tell You

The visits are short, mostly single-problem and written from case cards. They don't cover long follow-ups with five chronic conditions, inpatient rounds or behavioral health sessions. The reference notes are one clinician's style, so a different but correct note can look worse than it is.

Use the set to compare models and templates, set a baseline and catch regressions. Then have clinicians review notes from your own visits before you ship. The ambient scribe speech guide covers what the rest of the pipeline needs.

How Sully Fits

The Sully API takes a visit from audio to EHR-ready fields. Transcription handles recorded or live audio, Note Generation writes the note from your template, and Text to JSON pulls structured fields from it using a schema you define.

The Clinical Scribe Kit includes those schemas for 10 common US EHRs, with notes on how each one accepts documents from outside apps. So the same 207 ACI-Bench visits can test both the note and the fields you would write back to the chart.

To try it, get keys from the Sully API page and run the script above on five visits. Then open each visit's reference note next to yours and see what's missing.

Frequently Asked Questions

Where can I find doctor-patient conversation transcripts?

The Open Clinical Conversations dataset has 536 role-played doctor-patient transcripts and 1,701 short exchanges, gathered from ACI-Bench, PriMock57, the Fareez et al. OSCE interviews and MTS-Dialog. It's free under CC BY 4.0, with the Fareez part under CC0.

Are these real patient conversations?

No. Clinicians talked with actors, trained patients or staff working from case cards, and the visits were recorded and transcribed. There's no patient data in them, which is why they can be shared.

Which part should I use to test an AI scribe?

Use ACI-Bench for whole-note quality, MTS-Dialog for single sections, and PriMock57 when you also want to test transcription. The Fareez interviews have no reference note, so use them for transcription tests and for clinician spot checks.

Can I use the dataset commercially?

Yes. CC BY 4.0 and CC0 both allow commercial use. Credit the original authors for the CC BY parts; the attribution file has the wording and CITATIONS.bib has the citations.

Is ROUGE a good metric for clinical notes?

It's a useful trend line, not a quality bar. ROUGE counts shared words with the reference, so it misses wrong facts phrased in familiar words. Pair it with a check for missing and unsupported facts.

Do I need medical templates to use the data?

You need some structure for the note. The kit's 19 templates work with Sully's Note Generation API as they are, and the Sully template library shows more note formats by specialty.

Sources

1. Open Clinical Conversations on Hugging Face

2. Clinical Scribe Kit on GitHub

3. Yim et al., ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation, Scientific Data (2023)

4. Ben Abacha et al., An Empirical Study of Clinical Note Generation from Doctor-Patient Encounters, EACL 2023

5. PriMock57: A Dataset of Primary Care Mock Consultations (Korfiatis et al., ACL 2022)

6. Fareez et al., A dataset of simulated patient-physician medical interviews with a focus on respiratory cases, Scientific Data (2022)

7. Sully Note Generation docs

8. Sully Text to JSON docs

9. Sully transcription docs

10. Sully API page