When we started building CliniNote, we had reasonable experience with production NLP systems for non-clinical domains. We knew about tokenization edge cases, entity boundary disagreements, nested entity problems, and annotation noise. None of that prior experience fully prepared us for what clinical text actually looks like at the sentence level. Clinical NLP is a distinct problem, not a harder version of general NLP.
The gap between benchmark performance on curated clinical NLP datasets and performance on real-world EHR text is large and systematically underestimated. i2b2 datasets and MIMIC-based benchmarks are prepared, formatted, and often partially de-identified in ways that remove or regularize some of the most common sources of noise in raw EHR data. A model that achieves F1 of 0.88 on i2b2 NER may achieve 0.71 on notes pulled from a production EHR export at a single institution. We saw both ends of that range in our early testing, and understanding the sources of the gap shaped most of what we built.
Abbreviation disambiguation: the first major lesson
Clinical abbreviations are context-dependent in ways that non-clinical NLP work does not encounter. "MS" can mean multiple sclerosis, mitral stenosis, morphine sulfate, or mental status, depending on the clinical context of the note and the specialty of the writer. "SOB" almost always means shortness of breath, but appears in rare contexts meaning something else. "PE" means pulmonary embolism in the Assessment and Plan, physical examination in the Review of Systems section header, and physical exam in the Objective section of a SOAP note.
A simple lookup table disambiguation approach achieves roughly 80 percent accuracy on these cases. The remaining 20 percent requires section-level context. Our disambiguation model processes abbreviations in the context of their section type, the specialty documented in the note header, and adjacent clinical terms. Specialty detection from note metadata and section classification both need to work before abbreviation disambiguation can work reliably. We found we had to solve those upstream problems before we could trust abbreviation resolution at a level acceptable for clinical research use.
Negation: what the rules-based systems get wrong
NegEx and its successors are well-established clinical negation detection tools. They are also easy to get wrong in deployment when applied to real EHR text without modification. The original NegEx rule set was developed and validated on discharge summaries, which have relatively regular syntax. Applied to progress notes, which have much more variable structure and frequently use shorthand, NegEx as-written produces false negation errors in specific syntactic patterns that are common in shorthand but rare in discharge summaries.
Two patterns caused us consistent problems until we addressed them. The first is negation scope truncation in list constructs: "no fever, N/V, SOB" correctly negates all three terms, but "fever, N/V, no SOB" can be mishandled by rule systems that treat negation as spanning in only one direction from the trigger word. The second is pseudo-negation in differential diagnoses: "cannot rule out MI" is a statement of uncertainty, not confirmation, but "MI ruled out" and "no MI" are both negations that end up in structurally similar positions depending on documentation style. The semantic difference between these matters for endpoint identification.
We spent more time on negation boundary detection and scope resolution than on almost any other single component of the pipeline. The test cases that exposed failures were not exotic edge cases. They were patterns that appear dozens of times per 100 notes in a real patient population. Getting them wrong would have produced systematic false-positive or false-negative extraction errors large enough to invalidate research use cases.
Section segmentation and semantic weight
The same clinical entity mentioned in different sections of a note carries different meaning for research purposes. "Type 2 diabetes" in the Past Medical History section is a pre-existing condition. "Type 2 diabetes" in the Assessment and Plan is an active problem addressed at the visit. "Type 2 diabetes" in the Family History section is not the patient's diagnosis at all. An extraction system that does not segment notes into sections and assign different semantic weight to entity mentions by section will conflate these and produce incorrect attribution in the extracted output.
Section segmentation for clinical notes is harder than it sounds because section headers are not standardized. Notes written in different EHR systems use different header conventions, abbreviations, and formatting. "HPI," "History of Present Illness," "Chief Complaint and History," and "CC/HPI" are all section headers for the same section type in use across different institutions. Some notes use no headers at all and rely on SOAP structure implicitly. We built a section classifier that identifies section type from both explicit headers and structural context, and tested it against note exports from multiple institutions to verify that it generalizes beyond the training distribution.
The annotation ceiling problem
Any extraction system trained on human-annotated clinical text is bounded by the quality of the annotation. This is not a theoretical concern. It materially affects how to interpret internal performance metrics and how to set expectations with clinical research teams.
When two trained annotators disagree on whether a given entity mention in a clinical note represents a confirmed diagnosis or a rule-out diagnosis, the inter-annotator agreement (IAA) calculation captures that as a disagreement. If IAA on negation scope for a specific note type is 0.82 (a Cohen's kappa often cited as good agreement in clinical annotation), that means 18 percent of the time two trained annotators looking at the same note give different answers. A model trained on one annotator's labels and evaluated against another's will have an asymptotic performance ceiling set by that IAA, regardless of how well the model has learned the underlying phenomenon.
We observed this concretely in our early validation work. On straightforward entity types like medication names and lab result values, our models reached F1 scores close to the IAA ceiling within a few thousand training examples. On negation handling, uncertainty markers, and temporal attribution, we were consistently below the IAA ceiling even with more training data. Some of that gap reflects model capacity limitations. Some of it reflects genuine ambiguity in the annotation: cases where even expert annotators do not fully agree, and where a model should arguably return a low confidence score rather than a confident wrong answer.
Confidence calibration and its practical importance
A model that returns a confidence score of 0.9 should be correct roughly 90 percent of the time on cases it scores at that level. Most clinical NLP models are not well-calibrated in this sense out of the box, and the miscalibration is typically overconfidence: models that assign 0.9 confidence to cases they actually get right 75 percent of the time. The practical consequence for research use is that if you use a 0.85 confidence threshold for auto-accept and the model is overconfident by 15 percentage points, your auto-accepted cases are less reliable than the threshold implies.
We spent significant effort on post-hoc calibration of our confidence scores using a held-out validation set with documented ground-truth labels. The calibration curve we target is one where confidence scores correspond to actual precision within about 5 percentage points across the range from 0.7 to 0.99. Getting there required both temperature scaling on the model output and separate calibration for different entity types, because model confidence is better calibrated for some entity types than others.
What we are still working on
We want to be direct about where our pipeline has known limitations that we are still addressing. Cross-document entity resolution, the problem of linking the same entity across multiple notes in the same patient record, is an area where our current implementation is functional but not at the same accuracy level as single-document extraction. Linking a diagnosis mentioned in a note from November to a lab value that confirms it in a note from December requires entity resolution across documents, and our error rate on cross-document links is higher than our error rate on within-document extraction.
Rare disease terminology and subspecialty shorthand outside the primary specialties we trained on (internal medicine, cardiology, oncology, neurology) also show reduced performance. When a team working in a less common specialty asks us to extract from their note corpus, we run a benchmark on a sample before committing to a confidence threshold, rather than assuming the same thresholds will hold across specialty contexts. Honesty about performance by specialty is something we consider a basic requirement of operating in a research context where the output affects study data quality.
