When someone asks how accurate CliniNote is, the correct answer starts with a question: accurate relative to what, measured how, on which document types? Accuracy in clinical NLP is not a single number. It is a family of measurements that depend on the annotation schema, the annotator agreement rate, the text domain, and the entity types being extracted. Getting this right matters because a poorly constructed evaluation will tell you what you want to hear instead of what you need to know.
This article covers the mechanics of annotation quality assessment as we apply it in practice. The intended audience is research teams evaluating clinical NLP tools or building internal extraction pipelines, who need to understand what the metrics mean before they can interpret them usefully.
What F1 Score Measures and Where It Falls Short
F1 score is the harmonic mean of precision and recall. Precision measures how many of the entities a system extracted were correct. Recall measures how many of the true entities the system found. F1 balances both. Against a fixed labeled dataset, F1 is a reproducible and interpretable metric, which is why it is standard across NLP benchmarks.
The problem in clinical NLP is that the metric is only as meaningful as the dataset it is measured against. A system can achieve F1 of 0.92 on an internal benchmark assembled from notes where the annotation task was relatively clear (diagnoses with explicit ICD code-level granularity in the Assessment section of structured notes) and achieve much lower performance on notes from a different care setting, a different EHR template, or a different specialty, where the same entity type is expressed through abbreviations, embedded in narrative, or negated in ways the benchmark dataset did not represent.
The benchmark coverage problem is not solved by making the benchmark larger. It is solved by making the benchmark diverse. A 500-note benchmark that spans primary care, cardiology, nephrology, and oncology notes from multiple clinical sites will reveal performance variance that a 5,000-note benchmark from a single institution's cardiology department will miss entirely.
Inter-Annotator Agreement as the Ceiling
Before a model is trained or evaluated, the annotation dataset has to be built. Annotators, typically clinicians or trained clinical abstractors, read notes and label the entities: diagnoses with their assertion status, medications with their dosing information, dates, procedures, and so on. Where annotators disagree about whether an entity is present, what its boundaries are, or what its assertion status is, the resulting annotation represents a choice between conflicting human judgments rather than an objective truth.
Inter-annotator agreement (IAA) is the measure of how consistently two or more annotators produce the same labels for the same text. The standard metric for clinical NLP annotation is Cohen's kappa or its multi-annotator extension Fleiss' kappa, which corrects for chance agreement. For well-defined tasks with clear annotation guidelines, IAA kappa values above 0.8 indicate strong agreement and produce annotation of sufficient quality to serve as a reliable gold standard. For more ambiguous tasks, kappa values in the 0.6 to 0.75 range are common and represent a harder ceiling for model evaluation.
The IAA ceiling matters because no automated system can consistently outperform the human agreement rate on the same task. If two experienced clinical annotators agree on 85% of entity boundaries in a given corpus, then a model achieving 88% recall is either capturing genuine signal that the inter-annotator process missed, or it is producing false positives that coincidentally match one annotator's labels. High IAA is a prerequisite for a trustworthy gold standard, not a desirable bonus.
The Assertion Status Problem
In general-purpose NLP benchmarks, entity recognition usually means span detection: find the text that refers to a person, place, organization, or other category. In clinical NLP, the entity type is only part of the information needed. A diagnosis extraction system also needs to classify each mention by its assertion status: is this condition affirmed as present, negated, uncertain, historical, or referenced as a family history?
Assertion classification is harder to annotate than entity span detection, and IAA scores for assertion status tend to be lower than for span detection alone. The ambiguity in clinical text is often genuine. "No evidence of pulmonary embolism on CT" is a clear negation. "The patient's shortness of breath could be consistent with early heart failure" is uncertain. "He reports his mother had hypertension" is family history. "He had a prior MI in 2018" is historical. Each of these requires a different annotation label, and annotators with different clinical training backgrounds sometimes make different calls on borderline cases.
For RWE cohort definition, assertion status is not a secondary concern. Including patients where the diagnosis was mentioned as a differential but not confirmed, or patients where the condition was historical but not currently active, can substantially distort a cohort. A system that achieves high F1 on entity span detection but handles assertion classification inconsistently will pass aggregate benchmark evaluation while failing at the task RWE teams actually need it to perform.
How We Build and Validate Annotation at CliniNote
Our annotation process uses double annotation with reconciliation for any entity type being added to a new specialty or note type. Two annotators label a sample independently, then adjudicate disagreements with a third clinical expert reviewer. Before each annotation session, we run a calibration pass on a small set of notes where the correct labels are pre-established, which surfaces annotator drift and allows for guideline clarification before it propagates into the training data.
For new entity types, we do not proceed to model training until the IAA on the annotation sample reaches at least kappa 0.75 for entity spans and 0.70 for assertion status. Those thresholds are not arbitrary; they reflect the minimum agreement level at which the gold standard is reliable enough to train a model whose errors will be distinguishable from annotation noise. Below those thresholds, you cannot tell whether a model is making systematic errors or whether the training labels are inconsistent.
We also maintain a hold-out evaluation set that is kept entirely separate from training data and is never used for development feedback. Performance on that set is the number we report externally. The internally optimized benchmark numbers are higher, as they always are in any ML development workflow; reporting those would misrepresent what a new user should expect on their own data.
What to Ask When Evaluating a Clinical NLP Tool
When a vendor reports accuracy numbers for a clinical NLP tool, there are several questions that determine whether those numbers are meaningful. What document types were in the benchmark? What entity types were evaluated, and was assertion status included in the evaluation or just entity span detection? What was the IAA on the annotation dataset? Was the evaluation set held out or derived from the same distribution as training data? Were the notes from a single institution or multiple sites with different EHR systems and documentation styles?
Vendors who cannot answer these questions in detail are reporting benchmark scores that are more about marketing than measurement. Vendors who can answer them, including ones whose honest answers reveal limitations in specific domains, are the ones whose numbers you can actually use to calibrate expectations for your specific use case.
The appropriate response to a tool that achieves F1 of 0.82 on cardiology notes and 0.74 on oncology notes is not to reject the tool but to understand the domain gap and decide whether the oncology use case requires additional validation work before deploying at scale. The worst outcome is using a tool in a domain where you have no performance data, discovering the problem after generating a cohort for a study, and having to reconstruct the methodology under deadline. Getting the annotation quality question answered before deployment is always cheaper than addressing it afterward.
