From Data Deluge to Drug Insight: Purpose-Built AI as the Catalyst for Next-Generation Therapeutics

Unstructured clinical data (which doesn’t conform to structured fields like drop-downs or checkboxes) includes clinical notes, imaging reports, physician narratives, device data, and patient feedback and account for roughly 80% of US health records. These data provide a wealth of insights that can accelerate the development of effective drug therapies. Yet, we’re relying on people and generic AI to wade through the billions of clinical records created each year, 97% of which go unused. As drug developers rush to adopt AI, cracks in this approach are surfacing—from hallucinated findings (fabricated or inaccurate AI-generated content) to acronym mix-ups that threaten pharmacovigilance, real-world-data (RWD) studies, and discovery research. To use RWD to its full potential, pharma needs purpose-built, clinically trained AI that understands medical terminology, recognizes and preserves document context, and excels under human supervision.

Generic AI is falling short for pharma

Generic generative AI, including large language models (LLMs), are effective for broad, general-purpose tasks but can't meet the specialized demands of pharmaceutical development. They've been found to hallucinate, misattribute, or lose track of critical clinical details mid-document, also known as "missing middle-memory", where the model fails to reconcile earlier and later portions of a note. For example, misattributing two glasses of wine per week as physical activity (if only!) or skipping over the patient-reported symptoms in the middle of a doctor's clinical notes. This may partly be due to how these models are trained, with conversational sources like news articles and Wikipedia. This training further complicates their use in healthcare settings. It makes them ill-equipped to interpret the domain-specific formats of clinical tables or industry shorthand, where Pt might mean prothrombin time, physical therapy, or patient. Medical acronyms, too, pose an insurmountable challenge. LLMs can read "AS" as a preposition rather than as Aortic Stenosis and generate noise in pharmacovigilance workflows. While a human would understand the shorthand based on the surrounding context, generic AI has no idea what it means. Beyond these problems, generic AI is too expensive for routine tasks like data collection and extraction. And importantly, the lack of traceability – meaning the inability to follow how an output was generated – is a major red flag for regulatory bodies like the FDA, which require transparency to ensure patient safety and data reliability.

How domain-specific AI can close pharma's human-to-computer gap

Unlike generic LLMs designed for broad tasks, domain-specific AI is purpose-built to precisely interpret the unstructured, often messy clinical data that fuels pharmaceutical development. Trained on text that mirrors real-world inputs—millions of annotated notes, lab reports, and scanned PDFs—these models are optimized for clinical complexity. They use clinical context to disambiguate terms like 'Pt' and map to standards like SNOMED, MedDRA, or ICD-10. They're also more efficient than generic LLMs, relying on smaller, task-specific architectures and a fraction of the computing resources to extract labs, detect adverse events, match patients to trial data, and flag inclusion criteria. With sentence-level traceability, they even create audit trails essential for pharmacovigilance and regulatory review. Most importantly, domain-specific AI is already making an impact, powering 10× faster chart abstraction, real-time safety-signal detection, and cohort identification across billions of clinical notes, turning an overwhelming flood of data into actionable insights.

Domain-specific AI implementation and governance

Implementing domain-specific AI in pharma isn’t just about technical challenges. It's also about building and maintaining trust. Human oversight is key in this process. While AI can quickly find relevant data points, it's still up to a clinician or researcher to validate them before they influence patient care or drug development (e.g. preventing a misidentified adverse drug reaction from influencing trial decisions). This human oversight is a crucial part of the AI landscape, particularly in high-risk use cases like AI scribes for clinical notes or reading radiology labs, where it must be clear who owns a mistake. Importantly, adoption requires cultural change at the organizational and team levels. Teams must see AI as a tool to improve quality, not replace their hard work. With proper governance and human oversight, domain-specific AI can cut computing costs, enhance data fidelity, and accelerate access to insights from billions of documents to improve the quality of pharmaceuticals and the AI processes that contribute to them.

While generic LLMs struggle with clinical nuance and accuracy, decrease efficiency, and increase risks and costs, domain-specific AI can revolutionize the industry. Purpose-built AI models lean on context to translate unstructured data from billions of clinical documents and notes into actionable insights at the speed and scale today's pharma industry requires, and the payoffs are already visible. Researchers are experiencing ten-fold gains in data extraction, leaner computing requirements, and access to higher-fidelity data for modeling. The next step is clear. Pharma leaders should replace generic AI with clinically trained, purpose-built AI in high-value RWD pipelines and anchor their AI pipelines to robust human oversight. The sooner the industry embraces domain-specific intelligence, the sooner life-changing therapies can reach the right patients.