Healthcare AI is advancing faster than the regulatory frameworks, standards, and governance practices designed to guide its use. That creates a challenge for healthcare leaders: how do we responsibly adopt technology that can meaningfully improve care and operations when many of the rules governing it are still being developed?
This question was central to the conversations at the Coalition for Health AI (CHAI) Fall Symposium in Austin, which I had the opportunity to attend last week.
The Symposium brought together leaders from across healthcare, technology, regulation, and standards organizations to explore what it takes to move trustworthy AI from principles into practice. Discussions covered everything from validation and data provenance to workforce readiness, enterprise adoption, and ongoing oversight.
I found the smaller breakout sessions particularly valuable because they created opportunities to work through the difficult questions that arise when organizations begin deploying AI in real-world environments. How much validation is enough? Who is responsible when an AI system changes? What does meaningful human oversight look like? How do we know an autonomous agent is operating within its intended boundaries?
There are not yet universally accepted answers to many of these questions. But one message stood out to me: uncertainty should not become an excuse for inaction.
Healthcare needs to continue putting AI to work, testing new use cases, evaluating results, and learning from experience. The distinction we must make is between moving quickly and moving without discipline. Responsible innovation requires us to do both: advance adoption while strengthening the governance needed to support it.
Several themes from the Symposium reinforced what I believe healthcare organizations should be prioritizing today:
Risk Lives in the Use Case, Not Just the Technology
One of the most important ideas discussed was that AI risk must be evaluated in the context of how the technology is used. The same underlying AI model can present very different risks depending on the task it performs, the information it accesses, who relies on its output, and the consequences of an error.
An AI application extracting information from an administrative document does not necessarily require the same oversight as a system supporting clinical decisions or an autonomous agent executing actions within a healthcare workflow.
This distinction is important because governance cannot be effective if every AI application is treated identically. Organizations need risk-based approaches that reflect the actual use case, including what the system is authorized to do, where human judgment is required, and what happens when something goes wrong.
That starts with clearly defining the intended use. Before we can meaningfully evaluate an AI system, we need to understand the workflow it supports, the decisions it influences, the actions it can take, and the potential impact on patients, members, clinicians, and operations.
The more precisely we define the use case, the better equipped we are to establish appropriate safeguards without creating unnecessary barriers to innovation.
Governance Must Evolve Alongside the Technology
Much of the infrastructure healthcare organizations rely on today was not designed for AI systems that continuously evolve or operate with increasing autonomy. Procurement, contracting, implementation, workforce training, and ongoing oversight all need to adapt.
For example, procurement teams must understand not only what a technology promises to deliver, but also the evidence supporting its intended use. Contracts need to establish responsibilities for monitoring, reporting meaningful changes, providing access to evidence, and responding to failures. Implementation teams need clear workflow ownership, escalation procedures, and contingency plans when AI must be paused or overridden.
And, these considerations extend beyond new AI vendors. Organizations must also understand how AI capabilities are being introduced into products and services they already use.
The challenge is creating governance that can keep pace with innovation without becoming so burdensome that it prevents organizations from realizing AI’s benefits. That requires adaptable safeguards, clearly defined boundaries for pilots, proportionate review processes, and established triggers for reassessment as systems or use cases change.
Governance should be embedded in how AI operates, not added as a separate process after deployment. When designed well, it gives organizations the confidence to experiment, learn, and scale successful applications while identifying and addressing risks early.
Human Oversight Must Be Meaningful, Not Symbolic
The concept of keeping a human in the loop came up repeatedly throughout the Symposium, and for good reason. We often describe human oversight as though the presence of a person reviewing an AI output is sufficient to establish accountability.
In practice, effective oversight requires much more definition.
Who is responsible for the review? What information do they have access to? Do they have the expertise and training to recognize an error? What conditions require escalation, and do they have the authority to intervene?
These are especially important questions as healthcare organizations introduce increasingly sophisticated AI agents into operational workflows. A human reviewer who lacks the necessary context, competency, or authority may provide little meaningful protection, even when the process technically includes human approval.
We need to design oversight around the decisions being made and the risks involved. That includes defining responsibilities, ensuring reviewers have appropriate information, and evaluating whether the controls work. Human oversight should strengthen accountability, not simply create the appearance of it.
Validation Is a Continuous Responsibility
Another consistent theme was the need to move beyond treating validation as a one-time milestone before deployment. Initial testing is essential, but it cannot tell us everything about how an AI system will perform across different environments, populations, workflows, and changing conditions.
Responsible adoption requires ongoing evaluation throughout the technology’s lifecycle. Organizations need to monitor performance, establish thresholds for intervention, understand the impact of meaningful changes, and maintain evidence that safeguards are functioning as intended.
There is an important distinction between documenting that a control exists and demonstrating that the control operated. This becomes even more consequential with agentic AI, where systems may move beyond generating recommendations to initiating or executing actions.
Organizations need to know what an agent can do, what information it can access, what decisions require approval, and how its activity can be observed, interrupted, or corrected. These capabilities are foundational to maintaining control as AI becomes more deeply integrated into healthcare operations.
Trust Requires Evidence, Provenance, and Traceability
The Symposium’s discussions about trustworthy data extraction reinforced an important principle: accuracy alone is not enough. As AI transforms unstructured clinical and administrative information into structured data used across other applications and workflows, organizations need to understand where that information originated, how it was interpreted, and how the resulting output can be verified.
This is relevant across applications ranging from clinical decision support and trial matching to prior authorization and quality measurement. But, the same expectations should extend to AI-enabled workflows more broadly. Source provenance, model and policy versions, inputs, outputs, actions taken, and meaningful system changes all contribute to the evidence necessary to understand how a process operated at a particular point in time.
For healthcare organizations, this is about more than satisfying an audit. Traceability enables teams to investigate unexpected outcomes, identify where problems occurred, and continuously improve system performance. It also supports a more fundamental requirement for enterprise AI: organizations must be able to understand and maintain control over the intelligence operating within their environments.
Trust cannot depend solely on a vendor’s assurances. It must be supported by evidence that organizations can access, evaluate, and act upon.
Industry Standards Must Enable Responsible Adoption
One of the most encouraging aspects of the Symposium was the emphasis on translating industry collaboration into practical tools and common standards.
Today, healthcare organizations frequently evaluate AI solutions using different questionnaires, evidence requirements, and approval processes. Vendors face repetitive requests, while buyers struggle to compare technologies consistently.
Efforts such as CHAI’s AI Intake Playbook are important because they create opportunities to standardize how organizations collect and assess information about AI systems. Common approaches can reduce unnecessary duplication, improve transparency, and help organizations make more informed decisions. They also allow vendors to provide reusable evidence while recognizing that each deployment still requires evaluation within its specific environment and intended use.
We are beginning to see similar principles reflected in more formal frameworks. The Joint Commission’s voluntary Responsible Use of AI in Healthcare certification, for example, focuses on organizational practices involving governance, data management, risk and bias, validation, monitoring, transparency, and education.
An important distinction is that the certification evaluates how an organization governs and uses AI rather than certifying an individual AI product. I believe this reflects a necessary evolution in how the industry thinks about trust:
We cannot govern our way to trustworthy AI solely by approving products. We must also govern how those products are implemented, operated, monitored, and used to make decisions.
This requires shared responsibility across technology developers, healthcare organizations, clinicians, regulators, standards bodies, and the people ultimately affected by AI-enabled decisions.
Moving Forward Together
My biggest takeaway from the CHAI Fall Symposium is that trustworthy AI will not emerge from a single regulation, certification, or governance framework. It will develop through continued collaboration, practical implementation, and the willingness to learn and adapt as technology advances.
We should expect standards to evolve. We should expect new risks to emerge as AI capabilities expand. And we should be prepared to revisit assumptions and strengthen safeguards as evidence develops. But we should not wait for every question to be answered before pursuing the benefits AI can deliver.
At Autonomize AI, this perspective aligns with our approach to building enterprise-grade AI for healthcare. We believe organizations should be able to put AI to work today while maintaining visibility, traceability, human oversight, and control over how intelligence is applied within their operations. Governance and trust must be designed into the infrastructure supporting AI, allowing organizations to adapt as technology, regulations, and business requirements evolve.
I left the Symposium encouraged by the industry’s willingness to confront difficult questions and work toward practical solutions. I am particularly energized by the opportunity to contribute to the conversations, standards, and playbooks that will help shape responsible AI adoption across healthcare.
Ultimately, trustworthy AI is not the destination. It is what enables us to pursue the larger goal: a healthcare system that is more accessible, responsive, efficient, and affordable for everyone who depends on it. Achieving that goal requires us to keep moving forward, with the discipline to govern what we build and the willingness to learn as we go.
About the Author
Gina Collins is Chief Regulatory Officer at Autonomize AI, where she leads regulatory strategy, compliance, and enterprise risk management to support safe, scalable AI adoption in healthcare. With more than 20 years of executive leadership experience across payer, provider, and government healthcare sectors, she is known for driving operational transformation, strengthening governance, and translating complex regulatory requirements into practical solutions. Prior to Autonomize, Gina held leadership roles at the FDA’s Center for Devices and Radiological Health and a Fortune 13 global health services company, leading initiatives focused on regulatory transparency, operational risk, and compliance.
