Real-world evidence is having a moment. The 21st Century Cures Act, FDA's 2018 framework on real-world evidence, and a series of EMA reflection papers on RWE use in regulatory decision-making have combined to make RWE a credible input in drug development conversations. Teams that spent years producing RWE studies primarily for commercial purposes now find those same methods under regulatory scrutiny. That change in context raises the quality bar significantly.
This primer is written for clinical research teams who understand the basics of RWE conceptually but are trying to close the gap between "RWE as a category" and "RWE as a production-ready data pipeline." We are going to focus specifically on the data side: where RWE comes from, what determines data fitness, and where the gap between structured EHR data and what you actually need tends to show up.
What Counts as Real-World Data
Real-world data (RWD) is defined by FDA as data relating to patient health status or delivery of healthcare routinely collected from a variety of sources. The key word is "routinely": this is data generated in the course of ordinary clinical care, not data collected for research purposes. The most common sources are electronic health records, claims and billing data, disease registries, patient-generated data from wearables and apps, and observational data from pragmatic trials.
Each source has its own strengths and gaps. Claims data provides broad longitudinal coverage across payers and settings but captures billing codes rather than clinical findings. A diagnosis code on a claim tells you a provider submitted a bill using that code; it does not tell you whether the condition was active, suspected, ruled out, or coded primarily for reimbursement. EHR data provides richer clinical detail, including free-text notes, laboratory values, imaging reports, and medication records, but coverage is often limited to a single health system and completeness varies significantly by care setting.
Registries offer high-quality structured data for specific disease areas but are expensive to build and maintain, often missing patients who receive care outside the registry network. Patient-generated data from consumer devices is increasingly available but has different measurement properties than clinical observations, and the populations willing to share such data are not representative of most patient populations of research interest.
The Quality Bar for Regulatory Submissions
The FDA framework identifies several data quality criteria relevant to RWE: data relevance (does the data source capture the population, exposures, and outcomes of interest?), reliability (is the data collected and maintained in a way that supports valid inference?), and fitness for use (do the data have sufficient granularity and completeness for the specific research question?).
For regulatory purposes, the most common application is a single-arm trial with an external control arm, also called an externally-controlled trial or hybrid trial design. The external control arm draws on RWD to represent how patients with the same condition perform without the experimental intervention. FDA guidance on externally-controlled trials specifies that the external control data must come from a comparable patient population, cover the same time period where possible, and have outcomes defined and measured consistently with the interventional arm.
That last point, outcome consistency, is where unstructured clinical data creates the most friction. A MACE endpoint in a randomized trial is adjudicated by a blinded clinical events committee working from source documents. The same endpoint in an EHR-based external control arm is derived from diagnostic codes, clinical notes, and procedure records that were never intended to support endpoint adjudication. Demonstrating equivalence between those two measurement approaches requires explicit validation work, typically a manual chart review sample against NLP-derived outcomes.
The Structural Data Gap
Most research teams working with EHR data discover the same fundamental problem: the data they need to answer clinical questions is largely not in structured fields. Structured EHR fields reliably capture demographics, billing codes, and laboratory results ordered through the EHR system. They do not reliably capture clinical findings documented in notes, medication changes decided during encounters, procedure complications, or the nuanced condition chronology that determines who belongs in a cohort and who does not.
A concrete example: a study examining the real-world effectiveness of a heart failure medication needs to identify patients with heart failure as an active diagnosis, not just a billing history. ICD-10 code I50.x in the problem list is a starting point. But I50.x also appears on billing records for patients with resolved heart failure, for patients admitted for an unrelated condition where heart failure is a historical comorbidity, and for patients where the provider is ruling out heart failure. The structured code does not distinguish these cases. The clinical note often does, through phrases like "active systolic heart failure," "history of heart failure, currently compensated," or "heart failure exacerbation."
For research teams without an automated way to process note text, the standard response is to add more billing code filters (multiple codes in a window, paired with specific medication orders) and accept higher misclassification as a tradeoff for operational feasibility. The quality of RWE cohorts built this way depends heavily on how well those proxies perform, and that performance is rarely empirically validated against chart review in the specific population being studied.
Where RWE Studies Typically Fail the Quality Review
Having reviewed a number of RWE study designs at various stages, the failure points cluster in predictable places. The most common is outcome definition. Teams define outcomes using ICD codes because codes are available in structured data. Reviewers ask whether the coded outcome is clinically equivalent to the outcome as defined in the protocol. If the answer involves assumptions about coding sensitivity and specificity that were not empirically tested in the study population, that is a problem.
The second common failure is covariate capture. Confounders in observational studies need to be measured. If a key confounder (say, symptom severity at baseline, or the presence of a complicating comorbidity) is not reliably captured in structured fields, the study either omits it from the model (introducing unmeasured confounding) or asserts that it was captured through a code-based proxy (requiring validation). Neither is a good answer without empirical support.
The third failure is population definition. Including patients who do not actually have the condition of interest, or excluding patients who do, creates a cohort that answers a different question than the one stated in the protocol. The degree to which structured codes alone can define a study population depends heavily on the condition. For some conditions with strong code-outcome correlation, structured data alone may be adequate. For many others, especially conditions with heterogeneous presentation or documentation patterns, note text is the difference between a defensible cohort and a questioned one.
What This Means for Data Infrastructure
The practical implication for research teams is that the data infrastructure investments made for routine analytics do not necessarily meet the quality bar for RWE submissions. A data warehouse that supports business intelligence queries on structured fields is not the same as a pipeline that can extract, validate, and document clinical findings from free-text notes for regulatory purposes.
That gap is where we see the most demand from teams using CliniNote. They have structured data infrastructure. They can run queries. The problem is that the structured data cannot answer the question they are actually trying to answer, and the path to answering it runs through the clinical notes that nobody has had a scalable way to process. Extraction from notes is not a separate research activity from the core RWE workflow. It is the piece that makes the core workflow defensible.
We are not claiming that note extraction solves all RWE data quality problems. It does not address differential loss to follow-up, time-varying confounding, or the fundamental challenge of unmeasured confounders that are not documented anywhere in the EHR. Those are design-level problems that require methodological decisions, not better parsing. But for the specific failure modes that come from relying on billing codes as proxies for clinical realities that are actually documented in notes, extraction from unstructured text is the right tool, and it is now a tractable one.
