This is the second in our series leading up to HLTH USA 2026, where our team will share insights on the legal and business issues facing physician innovators, healthcare leaders, entrepreneurs, and investors.
Artificial intelligence (AI) vendor contracts in healthcare often draw on generic software as a service (SaaS) templates that were never built for protected health information (PHI) or regulated medical software. Here are five frequently missing clauses and why each matters.
1. Zero Data Retention
Zero data retention (ZDR) means the vendor’s AI model does not store, log, or use session inputs and outputs beyond the time needed to generate the response. A ZDR clause should confirm that the vendor does not retain patient data submitted to the tool after processing, log it beyond a short, defined window, or use it to train or fine-tune its models. Without it, prompts containing PHI could sit in vendor logs indefinitely or feed a shared model. Many consumer-grade AI application programming interfaces (APIs) retain data by default unless an enterprise opt-out is negotiated, so get the vendor’s default setting in writing.
Make sure the restriction reaches the entire service, including subprocessors, diagnostic logs, backups, and any human review of prompts or responses. If the vendor needs a limited retention exception for security or troubleshooting, define the permitted data, purpose, access controls, and deletion deadline rather than accepting a blanket carve-out. Ask for written confirmation that the vendor has enabled the agreed settings for your account and cannot change them without your approval.
2. Business Associate Agreement and Minimum Necessary Use
Any AI vendor that creates, receives, maintains, or transmits PHI needs a signed business associate agreement (BAA) before data flows. Enforcement actions confirm that disclosing PHI to a vendor, including a software company using PHI to operate its service, without a BAA is itself a Health Insurance Portability and Accountability Act (HIPAA) violation. The BAA should also build in the minimum necessary standard, limiting disclosures of PHI to business associates to the minimum amount reasonably necessary for them to perform their function. For an AI vendor, that means scoping exactly which data fields it can access, not a full data feed by default.
Do not let a signed BAA substitute for reviewing the vendor’s actual data practices: align the BAA, service agreement, and product settings on permitted uses and disclosures. Address whether subprocessors can access PHI and require contractual protections that follow the data through that chain. Before launch, have privacy and technical teams confirm that the integration sends only the approved fields and that any broader access requires a separate review.
3. Breach Notification Timelines
Generic agreements often bury breach notice in a vague “commercially reasonable efforts” standard. A business associate must notify affected covered entities following discovery of a breach of unsecured PHI. Convert that duty into a specific contractual deadline (for example, within 24 to 72 hours of discovery), with an obligation to cooperate on the covered entity’s own notification duties and cover reasonable response costs.
Specify who receives the notice, how the vendor must deliver it, and what the initial report should include, such as the systems involved, potentially affected data, and immediate containment steps. Require prompt preliminary notice even if the investigation is incomplete, followed by regular updates as facts develop. Address security incidents separately from confirmed breaches so the vendor cannot postpone an operational alert while it decides whether an event triggers formal breach notification.
4. Model and Algorithm Change Control
AI models are frequently retrained or silently updated. A clause requiring advance notice of material model changes and the ability to test or reject an update before it goes live protects against unannounced shifts in output quality or clinical risk (especially for tools touching diagnosis, triage, or treatment recommendations).
Define a material change broadly enough to capture changes to the underlying model, intended use, performance thresholds, and connected data sources, not just a new product version. Require documentation of validation results and give the provider a meaningful testing period before deployment into clinical workflows. If an update fails agreed acceptance criteria, preserve the ability to roll back, suspend the affected functionality, or terminate without an early termination penalty.
5. Output Accuracy and Clinical Liability Allocation
Vendors routinely disclaim all responsibility for AI output accuracy, pushing full clinical liability onto the provider. Negotiate clear language on what the tool is validated to do, require indemnification for defects in the model or training data, and confirm the agreement does not quietly turn the software into an unregulated substitute for a cleared medical device.
Translate validation claims into measurable commitments tied to the intended patient population, clinical setting, and agreed use case, with a process for reporting and correcting unsafe or unreliable outputs. Separate the provider’s responsibility for clinical judgment from the vendor’s responsibility for software defects, failure to meet agreed specifications, and unauthorized changes. Review liability caps, exclusions, and insurance alongside those commitments so a broad disclaimer or a nominal cap does not undercut the protection you negotiated.
Practical Takeaway
Run every AI vendor contract through a checklist built around these five areas, not just standard data security boilerplate. A short data flow diagram showing where patient data goes, who touches it, and whether it is retained is often the fastest way to surface gaps the vendor’s standard paper does not address.
Bring legal, privacy, information security, and clinical stakeholders into the review before a pilot begins, and assign an owner to verify that negotiated safeguards are implemented. Revisit the checklist when the vendor changes its model, adds a subprocessor, or expands the tool’s intended use, rather than treating contract signature as the end of the review.
I’ll be joining my Husch Blackwell colleagues at HLTH USA 2026 in Las Vegas, where we’ll be presenting as part of the Physician Innovator Series on November 15. We’ll be discussing these issues and other legal and business considerations facing physician entrepreneurs, healthcare leaders, and investors.
If you’re attending HLTH, I’d welcome the opportunity to connect, hear what you’re working on, and discuss some of the challenges and opportunities you’re seeing.