Introduction

When a new model scores higher on a public benchmark, how do we actually know it will deliver improved results to clinicians? An evaluation is only useful if it accurately represents the setting where a model will be used, and clinical settings vary widely in patient population, specialty, visit type, and regional or organizational requirements.

We’ve found that current methods for evaluating clinical documentation, and clinical AI systems more broadly, are unreliable, inefficient, or impossible to audit. Expert clinician review remains the gold standard, but it cannot be repeated for every change to a model, prompt, or harness. Few public benchmarks are built on real-world clinical documentation, and those that do rarely reflect real deployments. LLM judges, without careful design, miss information and give inconsistent verdicts. The definition of ground truth is often very nuanced, as two clinicians may prefer different outputs even when presented with the same clinical encounter.

Without representative evaluations, the quality of generated clinical content cannot faithfully be measured. Inaccuracies in documentation can propagate to downstream systems, including billing, patient summaries, and referrals. They can affect clinician trust and can limit adoption of the system. Conversely, if clinicians begin to rely on the technology more, the rate of uncaught errors can even increase.

We believe that better evaluations lead to better clinician experiences. High-quality evaluations allow us to understand current failures and build actionable targets for optimization. Improvements made mean that clinicians can spend less time correcting information and return more time to providing care. Here, we present Sully’s framework for evaluating the accuracy of AI-driven clinical documentation, and how this has led to improved experiences for our clinicians. These principles are the driving force behind how we build evaluations across our suite of clinical AI systems.

Evaluating Accuracy During Development: Offline Evaluations

Offline evaluations measure how well a system performs before it is deployed. We use offline evaluations to evaluate every change to a prompt, model, or harness before it is used by clinicians.

An offline evaluation runs a candidate system on a fixed set of representative patient encounters, then scores the resulting notes against encounter-specific rubrics for clinical accuracy and safety. Because the encounters and rubrics stay the same, any difference in scores comes from the change itself, which lets us compare versions directly.

Testing on a fixed task set with LLM-based scoring is now standard practice and well documented. We believe that what determines the true value of offline evaluations is how well they represent real production data. As a result, we’ve built out a comprehensive pipeline to generate representative data for offline evaluations.

Constructing Representative Encounter Sets

A useful benchmarking dataset must do two things: reflect the distribution of encounters clinicians actually see, and include intentionally difficult cases where documentation is most likely to fail. To build this, we embedded and clustered de-identified production transcripts, then separately clustered the de-identified notes generated from them. Comparing the two cluster maps shows where similar visits produce consistent notes and where they diverge. From these maps, we select encounters in three groups:

• Common cases: Representative samples from each cluster, which keeps the dataset grounded in the real-world distribution.

• Divergent case pairs: Pairs of encounters whose transcripts sit close together in the embedding space but whose notes land in different clusters. These represent nuanced or complex cases, where a small detail can change the correct note in a critical way (e.g., a medication that was discussed but not prescribed). A model that misses these produces notes that read well but are clinically incorrect.

• Edge cases: Encounters that fall outside every transcript cluster, along with visits at the extremes of length. Long, multi-problem visits test thoroughness and capture, while very short or nearly empty visits test abstention.

Selected cases serve as blueprints for generating synthetic encounters that preserve the structure, specialty, visit type, number of problems, and specific complexities. This process enables us to supplement the dataset as the distribution of encounters changes. We additionally build targeted sets for specific failure patterns, which let us improve one behavior while checking that others do not regress.

Rubrics and Scoring

Once an encounter set is fixed, we build an encounter-specific clinical rubric for each case in the dataset. Each rubric contains binary, checkable requirements covering atomic pieces of information that must be captured correctly. These include:

• Inclusion: Capturing clinically important information supported by the encounter transcript and supplied context.

• Exclusion: Avoiding unsupported additions, contradictions, and misleading statements.

• Uncertainty and nuance: Preserving uncertainty, hedging, attribution, timing, and treatment status—for example, distinguishing a medication discussed as a possibility from one actually prescribed.

The requirements for rubric construction and verification are built with clinician input. To continually generate and refine rubrics, we use a multi-step process of deterministic steps and LLMs, wherein reference answers are generated, adjudicated, and compiled into binary rubric items. Rubric requirements are sorted into tiers of safety importance, with mandatory requirements being most critical.

Representative examples of rubric items. Each rubric item is a binary yes/no question.

After rubrics are built and fixed, we test candidate models on the evaluation dataset. Each candidate model generates the documentation, and every result is scored against its encounter-specific rubric. Scoring uses multiple large reasoning LLMs from different model families, which assign verdicts with justification and calculate the safety-weighted pass rate. This metric penalizes failures on more important rubric items; in particular, any failure in a mandatory requirement results in a fail for the rest of the requirements in the note. Across the LLMs we use for judging, we find an agreement rate of 99.04% and Cohen’s κ of 0.928.

Clinical rubric scores on one of our evaluation sets for women’s health. Prompts, harnesses, and settings were held equivalent across models.

The Impact of Offline Evaluations

Improvements on our offline evaluations have translated into measurable gains in production. Offline evaluations allow us to rapidly test a variety of changes against a fixed encounter set before rollout. For one women’s health clinic, we used offline evaluations to test and optimize a completely new harness architecture tuned to their requirements. Once we deployed the selected architecture, continuous edit stretches fell by 20.7% per note. For another customer, we employed offline evaluations as we tested improvements driven by low ratings and clinician feedback. We validated a model upgrade on offline datasets and raised CSAT by nearly 30% after deployment. In both cases, improvements validated offline resulted in measurable gains for clinicians after deployment.

Evaluating Clinical Accuracy in Production: Online Evaluations

Online evaluations measure the actual performance of an AI system after production release. Documentation preferences and requirements of our models vary widely by region, organization, site, and even by clinician. Our goal with online evaluation is to accurately measure failures and turn them into actionable signals. There are a number of existing methods of doing this, which we first tried:

• Explicit feedback: CSAT, comments, support tickets.

• Edit-based metrics: tracking corrections to the note and computing text overlap or similarity scores.

• Transcript-based evaluation: using an LLM judge to score a generated note against its transcript.

Explicit feedback is valuable, but it covers a small fraction of notes, varies widely, and misses signal. Edit-based metrics fail to accurately characterize the fact that the size of an edit is not related to its clinical importance. Single pass LLM-as-judge was inconsistent and gave diverging verdicts on similar inputs.

We therefore built a system that measures every edit a clinician makes, breaks it into atomic pieces of information, and uses each one to identify the error in the generated note: fabrication, omission, misinterpretation, or departure from the clinician’s preferred style. Each failure is also assigned a type (e.g., medications, plan items, diagnoses) and a severity (critical, high, medium, or low).

This process and taxonomy support our continual improvement process in multiple ways. They allow us to trace a reported problem to its root cause and confirm that the fix resolved it, monitor each deployment for unreported regressions, and measure the impact of requested changes to the system.

Identifying Fabrications, Omissions, and Misinterpretations

We treat online evaluation as a three-way comparison between the generated note, the final clinical record, and the visit transcript and context. We first compare the signed note with the generated note and split the differences into individual edits. Each edit is described by whether the change added, removed, or modified content; whether it concerns clinical content; and whether it is supported by the encounter information. By reconciling this information, we can construct a rich error taxonomy.

Simplified table showing how errors are characterized

This classification runs as a multi-stage pipeline. Each stage combines deterministic steps with LLM reasoning, and LLMs are used only when they are deemed necessary in ablation testing. LLM steps use two judges from different model families. For reruns on the same initial notes with the same transcriptions, the agreement between the final issues is 96.3%.

Validating Online Evaluations

Online evaluations produce signal on the error of individual issues, which can be rolled up into general trends of our AI systems. We’ve found that online evaluations produce findings consistent with explicit clinician feedback on the same notes and can also detect issues that clinicians have not raised. For example, in one of our deployments, we detected that the system would over-assert in specific scenarios where a plan was suggested but not confirmed, and optimized towards abstention instead. In another case, we detected that a certain clinician would tend to restructure notes and were able to set up a custom template for them in response.

Online evaluations provide an invaluable way to measure how well improvements to the system actually do in production. The following shows a case study for an ambulatory care clinic serving remote villages. Online evaluations helped measure improvements during a simultaneous overhaul of custom note templates, changes to documentation requirements, and a migration to a new model. By characterizing clinician edits at baseline, we were able to understand the patterns of fabrications and omissions, and were able to optimize the system in response and measure outcomes. After these changes, fabrications fell by 83.1%, omissions by 69.1%, and misinterpretations by 87.1%.

We apply this process to every deployment independently, which helps us understand failures and improvements at the macro and micro levels. Efficient error capture and triage mean that the process can effectively provide signal from large organizations to family clinics.

Conclusion

How Better Evaluations Lead to Improved Safety

Our approach to evaluations is guided by the principle that they must be representative as well as actionable. Our evaluation loop connects these two steps. Online evaluations detect failures, offline results guide research and engineering changes, and online analysis measures their effect after deployment.

LLMs make these evaluations scalable, but we use them under strict constraints to keep results reliable and avoid circularity. We use an LLM only where a step requires judgment and a deterministic method performs worse. Important judgments are made by independent judges from different model families, and every verdict must include a justification that can be audited.

Building faithful evaluations is an ongoing process, and we are consistently testing new methods of evaluation. As an example, as clinicians come to trust clinical AI tools more, they may review notes less closely, and errors that would once have been caught can reach the signed record. Edit-based analysis cannot detect such errors, so we are already deploying online rubric generation alongside edit-based analysis to measure note quality independently of clinician review.

Ultimately, the safety of AI systems in clinical settings depends on whether their errors can be found, measured, and reduced. Evaluations are the most important lever in enabling this process. As AI continues to become embedded more and more in clinical care, evaluations matter as much as the models themselves.