This is the second in a series of blog posts focused on AI and clinical trials/research space, highlighting topics to be discussed at the upcoming BIO International convention on June 22-25, 2026.
Introduction
Your latest trial sponsor is using AI to accelerate protocol development and patient recruitment,[1] cutting timelines that once took months down to weeks. The efficiency gains are undeniable. Then your quality assurance team asks a simple question: “Can you reproduce these AI-generated inclusion criteria?” You realize you can’t. The same prompts produce different outputs each day. When you ask the AI vendor how to validate their tool under ICH E6(R2), they send you marketing materials about “cutting-edge technology,” not validation documentation.
You’re now facing an FDA inspection with protocols you cannot fully explain and recruitment algorithms you cannot audit. The sponsor remains fully liable for compliance, but the AI has diffused the decision-making process without reducing legal responsibility. How do you validate tools that produce different outputs each time you use them?
The fundamental problem is asymmetric accountability. Sponsors remain fully liable for protocol quality, enrollment diversity, and regulatory compliance, yet AI tools diffuse the decision-making process across opaque algorithms and vendor systems. When an AI-assisted protocol fails FDA review or a recruitment algorithm produces demographically skewed enrollment, the sponsor answers to regulators. The AI vendor does not.
Unlike traditional software validation challenges, AI introduces risks most sponsors haven’t anticipated: protocols that cannot be reproduced, recruitment algorithms that systematically exclude populations, and black-box decision-making that defies FDA audit trail requirements. This article examines two critical risk areas: protocol design risks and algorithmic bias in recruitment and identifies the validation and contracting frameworks needed to address them. Understanding these risks now—before they become audit findings or warning letters—is the difference between strategic AI adoption and compliance crisis management. We begin with the validation challenges that emerge when AI enters protocol development.
Section 1: Protocol Design Risks
Sponsors are discovering that protocols drafted or substantially influenced by generative AI tools create validation challenges under ICH E6(R2)’s computerized systems requirements, which were designed for deterministic software that produces reproducible outputs.[2]
The Reproducibility Problem
Unlike deterministic software that produces identical outputs from identical inputs, large language models generate variable responses. AI outputs are typically non-deterministic, meaning the AI may exhibit a range of behaviors under the same input conditions.[3] A sponsor using AI to draft inclusion/exclusion criteria today may receive different recommendations tomorrow from the same prompts. This non-deterministic behavior creates tension with ICH E6(R2) requirements for validated computerized systems.[4] How does a sponsor validate a protocol development tool that cannot guarantee reproducible results?
AI Hallucinations in Clinical Design
Generative AI tools confidently produce plausible-sounding but factually incorrect content—the “hallucination” problem.[5] Even the FDA’s internal AI assistant, Elsa, has generated false citations and fabricated non-existent studies, prompting FDA officials to warn that outputs are “unreliable” without human verification.[6] In protocol development, this manifests as citations to non-existent studies, incorrect dosing regimens, or flawed statistical approaches.
Regulatory Acceptance and Audit Trail Deficiencies
Explicit standards for validating AI-assisted protocol development tools remain underdeveloped. The FDA’s January 2025 draft guidance on AI in drug development addresses AI use for data analysis, manufacturing, and clinical endpoints, but does not specifically cover protocol design or drafting.This creates regulatory uncertainty about what validation documentation will satisfy FDA expectations during inspections.
The documentation challenge is particularly acute, given that up to 50% of clinical trial datasets contain errors or inconsistencies requiring extensive cleaning processes.[7]Regulators expect complete audit trails showing who made protocol decisions, when, and why.[8] AI-assisted protocols create black boxes. When an AI tool recommends a specific primary endpoint or sample size calculation, the reasoning often cannot be fully explained or documented. This creates untenable situations during FDA inspections when investigators ask sponsors to justify protocol design choices substantially influenced by opaque algorithms. Critically, these validation and documentation challenges should be addressed not only through internal processes but also through carefully negotiated AI vendor contracts. Sponsors should ensure their agreements include audit rights allowing inspection of model architecture, version control requirements identifying which model version generated specific outputs, and cooperation obligations requiring AI vendors to participate in FDA meetings and provide validation documentation.[9]
Section 2: Algorithmic Bias in Recruitment
AI-powered patient recruitment tools promise to accelerate enrollment by mining electronic health records, claims databases, and patient registries to identify eligible candidates.[10] Studies demonstrate that AI recruitment tools can reduce overall trial costs by up to 40%, making them attractive to sponsors.[11]But these same algorithms can systematically exclude populations in ways that violate regulatory expectations and compromise scientific validity.
How Bias Enters Recruitment Algorithms
AI patient-matching tools learn from historical data with well-documented diversity problems. Despite decades of efforts to broaden participation, “certain groups continue to be underrepresented in many clinical trials,” with populations including racial and ethnic minorities, older adults, children, and those with certain comorbidities frequently excluded “without strong clinical or scientific justification.”[12] When algorithms train on such datasets, they perpetuate these patterns through algorithmic bias, producing systematically incorrect results due to limitations in the training data.[13]
Regulatory and Scientific Implications
The FDA’s December 2025 guidance on enhancing participation in clinical trials requires sponsors to “enroll participants who reflect the characteristics of the intended-use population with regard to age, sex, race, and ethnicity.”[14] The guidance warns that inadequate participation can lead to “insufficient information pertaining to medical product safety and effectiveness for product labeling.”[15] AI tools that systematically produce homogeneous enrollment patterns create regulatory risk, potentially triggering FDA questions during Investigational New Drug application reviews or complete response letters citing inadequate diversity data.
Beyond FDA oversight, algorithmic bias raises significant equity and scientific concerns. Researchers have identified data management and algorithmic design as critical means of “identifying and mitigating bias and promoting health equity” in clinical research.[16] The FDA has documented that differences in response to medical products have been observed in racially and ethnically distinct subgroups, attributable to genetic, metabolic, environmental, and sociocultural factors.[17] AI-driven enrollment bias amplifies this risk. Drugs approved based on homogeneous trial populations may show different safety and efficacy profiles in real-world diverse populations, potentially leading to post-market safety signals, label changes, or market withdrawal—consequences that may not surface until years after approval, when patient harm has already occurred.
Conclusion
Protocol design and patient recruitment are where AI-related compliance gaps are emerging first. Sponsors assuming AI tools are “trial-ready” face gaps during FDA inspections. These gaps cannot be closed through internal validation alone; they should also be addressed through AI vendor contracts that include audit rights, performance warranties, version control requirements, and indemnification for AI-specific harms. For detailed guidance on negotiating these provisions, see our companion analysis of AI vendor contracting.
Successful sponsors validate reproducibility before Institutional Review Board submission, audit algorithms for bias before enrollment, secure contractual protections from AI vendors, and build documentation that withstands scrutiny. The question isn’t whether AI will transform clinical trials—it’s whether sponsors will manage the transformation proactively or explain failures retroactively.
[1] Clinical trial sponsors are integrating artificial intelligence into protocol development and patient recruitment. See Askin S, Burkhalter D, Calado G, El Dakrouni S. Artificial Intelligence Applied to clinical trials: opportunities and challenges. Health Technol (Berl). 2023;13(2):203-213. doi: 10.1007/s12553-023-00738-2.
[2] ICH E6(R2) Good Clinical Practice: Integrated Addendum to ICH E6(R1), Section 5.5.3 (requiring that computerized systems used in clinical trials be validated to ensure accuracy, reliability, consistent intended performance, and the ability to discern invalid or altered records).
[3] National Institute of Standards and Technology, “Challenges to the Monitoring of Deployed AI Systems,” NIST AI 800-4, at 1 (March 2026) (“AI outputs are typically non-deterministic, meaning the AI may exhibit subtly or even vastly different behavior under the same input conditions.”)
[4] ICH E6(R2) Good Clinical Practice: Integrated Addendum to ICH E6(R1), Section 5.5.3 (2016).
[5] Adam Tauman Kalai et al., “Why Language Models Hallucinate,” arXiv:2509.04664v1 [cs.CL] at 1 (Sept. 4, 2025)
[6] Chris Mazzolini & Mike Hollan, “FDA’s Elsa AI Tool Raises Accuracy and Oversight Concerns,” Applied Clinical Trials (July 23, 2025), (quoting anonymous FDA officials describing how Elsa “was making stuff up” and citing studies that don’t exist).
[7] David B. Olawade et al., “Artificial Intelligence in Clinical Trials: A Comprehensive Review of Opportunities, Challenges, and Future Directions,” International Journal of Medical Informatics 206 (2026) 106141, at 2.
[8] FDA, “Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products,” Draft Guidance (Jan. 2025).
[9] For detailed guidance on AI-specific contract provisions that address these protocol development risks, see our analysis of AI vendor contracting for clinical trial sponsors. Kappel, K., & Mannebach, M. (2026). AI in the Fine Print: Why Your Contracts May Not Be Ready, Husch Blackwell Healthcare Law Insights. https://www.healthcarelawinsights.com/2026/05/ai-in-the-fine-print-why-your-contracts-may-not-be-ready/
[10] Scott Askin et al., “Artificial Intelligence Applied to Clinical Trials: Opportunities and Challenges,” Health and Technology 13:203–213, 209 (2023)
[11] David B. Olawade et al., “Artificial Intelligence in Clinical Trials: A Comprehensive Review,” Int’l J. Med. Informatics 206 (2026) 106141
[12] FDA, “Enhancing Participation in Clinical Trials — Eligibility Criteria, Enrollment Practices, and Trial Designs,” Guidance for Industry at 2 (Dec. 2025) (noting that “certain groups continue to be underrepresented in many clinical trials” despite ongoing efforts).
[13] Askin et al., supra note 6, at 212 (noting “underrepresentation of certain populations in datasets could lead to overfitting… which can impact the AI model performance in the underrepresented population”).
[14] FDA, “Enhancing Participation in Clinical Trials — Eligibility Criteria, Enrollment Practices, and Trial Designs,” Guidance for Industry at 2, 6 (Dec. 2025)
[15] Id.
[16] Askin et al., “Artificial Intelligence Applied to Clinical Trials: Opportunities and Challenges,” Health and Technology 13:203–213, 212 (2023).
[17] FDA, “Enhancing Participation in Clinical Trials — Eligibility Criteria, Enrollment Practices, and Trial Designs,” Guidance for Industry (Dec. 2025).