.png)
AI is rapidly reshaping how real-world data (RWD) is generated and analyzed across healthcare and life sciences. As AI becomes embedded in evidence generation, regulatory discussions, and access decisions, questions of trust, transparency and accountability play an increasingly important role.
In this interview, Michael Fronstin, GM of syndicated platform at HealthVerity, discusses these challenges with Oracle compliance expert, Jessica Santos, PhD, AIGP. Their conversation explores what responsible AI guardrails mean in practice and how governance and transparency can enable innovation without compromising credibility or compliance.
What AI guardrails really mean in healthcare and life sciences
Michael Fronstin: AI in healthcare is moving fast, but trust is fragile. When you hear leaders talk about AI guardrails, what do you think they specifically mean and often misunderstand or oversimplify?
Jessica Santos: There is no doubt AI is a very powerful tool, but it also presents a lot of risks, such as hallucination, bias, confidentiality, privacy, lack of transparency, and no accountability of the output, etc. So it is essential we need to have guardrails in place to mitigate these risks while harvesting the power of AI.
Michael Fronstin: Of the risks you mentioned, which are the most important and impactful? And can you share why?
Jessica Santos: That is a tough question. Every one of the risks outlined above can invalidate RWE efforts, undermine scientific integrity, and ultimately harm patients, physicians, and payors who rely on RWE for critical decision-making. They can erode trust with data subjects, damage the reputation of companies and stakeholders, lead to substantial financial penalties (including fines of up to 7% of annual revenue), and in severe cases even result in criminal liability.
The biggest mistakes organizations make when applying AI to real-world data
Michael Fronstin: From your perspective, what are the most common and most dangerous mistakes organizations make when deploying AI with real-world data?
Jessica Santos: Without understanding the risks of AI and trusting it regardless will certainly top the list. For example, uploading real-world data into an OpenAI system could compromise confidentiality, intellectual property, and the privacy of data subjects. Likewise, relying on AI-generated outputs without appropriate human oversight risks undermining scientific integrity and jeopardizing the validity, transparency, and replicability of the work.
Why transparency in data sourcing and modeling matters more than perfection
Michael Fronstin: You have said that opacity is often riskier than imperfection. Why is not knowing how data is sourced, linked, or modeled more dangerous than working with incomplete data?
Jessica Santos: Our whole RWD industry is based on trust and accuracy of the data, analysis, and conclusion that follows. Data validity and replicability are not optional. Our RWD scientists do not want a creative AI, or funny or amuse us, we want 100% trust in how the data is sourced and verified (not synthetically generated); we will have data management plan to treat missing and incomplete data with different stakeholders and approval process, not randomly ‘fixed’ by AI; and we develop statistical analysis plan for data linking and modelling with a group of experts agree on methodologies and algorithms, not have an ‘opaque AI’ provide different output every minute.
Michael Fronstin: I agree on the importance of your point. Given these RWD often result in evidence submitted to various regulatory agencies and payers, the data foundation is the starting point. Transparency has often been an issue when working with various consulting organizations and something which has prevented many researchers from using AI tools. They often say it’s a black box or that they don’t really know how the tool is working or get to see the code. In order to build the trust you mentioned, the cost of entry is moving toward tools which provide full transparency into the code or what’s now being coined as under the glass.
Essential AI governance guardrails for credible and defensible RWE
Michael Fronstin: For teams in RWE, Medical Affairs, Epidemiology, or Market Access, AI outputs increasingly inform evidence generation, regulatory discussions, and payer decisions. What guardrails in addition to transparency are essential to ensure those insights remain credible and defensible?
Jessica Santos: This topic follows perfectly from what we just discussed. There are several things we need to do:
Choose an AI development or deployment environment – open AI may be OK for general knowledge with human verification, but enterprise or on-premises development will be recommended for any evidence processing or generation that involves confidential or personal information. Verify the source data and work with data providers or data controllers for terms of use and data rights. If AI is used for data cleaning, annotation, processing, or labeling, we need to fully document the algorithm and settings and ensure traceability. Ensure AI model accuracy and replicability. We can deploy champion and challenger models, for example. If we ask the same question twice, will AI give us the same answer? What if we ask the same question slightly differently? Accountability. Who is responsible for AI selection, settings, deployment, and results credibility, and how do we ensure human in the loop (HITL)? Finally, documentation. Every step must be documented for governance, audit trail, and regulatory inspection.
How real-world consequences raise the bar for responsible AI use
Michael Fronstin: Healthcare and life sciences carry real-world consequences, from patient outcomes to reimbursement and policy decisions. How does that reality raise the bar for responsible AI use?
Jessica Santos: ‘Human in the loop’ is a phrase we hear very often in AI governance. Why do we need ‘human in the loop’? Because we need someone to be accountable, especially when it comes to cases that have real-world consequences. If an AI sings me a song I don’t particularly like, it has completely different consequences than providing wrong treatment options, health outcomes, or reimbursement or policy decisions. The latter could be potentially devastating. We have witnessed that in other sectors such as autonomous vehicles.
Michael Fronstin: When you say HITL should be accountable, how do they do this? What actions can the human in the loop take or consider taking to avoid hallucinations and misalignment or mistakes that can have real consequences?
Jessica Santos: A responsible workflow should begin with a careful review of the data before it is introduced into any AI system. That means removing confidential, personal, or sensitive information unless it is absolutely essential. If such data must be included, an accountable person should apply appropriate safeguards, such as differential privacy, noise injection, or masking, to protect individuals. At each stage of the process, the system’s behavior should be examined: verify that the algorithm is functioning as intended, ensure error levels are acceptable, and document any assumptions or limitations. Finally, the AI’s output should never be accepted at face value. A human should review, validate, and formally approve the results. These are just a few of the key steps in a responsible HITL AI adaptation.
How strong AI governance enables faster innovation and scale
Michael Fronstin: Many leaders worry governance will slow innovation. Based on your experience, how do strong guardrails actually help teams move faster and scale with confidence?
Jessica Santos: Great question. Governance and compliance are often regarded as barriers to innovation throughout history. Who wouldn’t like to innovate without any boundaries? But at the same time, we want to live in a world where our food is safe to eat (regulated); the building we are in won’t collapse (inspected); the road we drive on is secure and governed. It is always that the company provides regulated products and services will gain customer and industry trust, not the other way around.
Michael Fronstin: Is there anything to add on what life sciences companies do to help move faster or scale? Are there any generic use cases or hypotheticals you can share?
Jessica Santos: Drug discovery in the life sciences can take a decade or more, and nearly 90% of candidates ultimately fail. With stakes that high, the last thing anyone working in RWE wants is to produce analyses that regulators, clinicians, or patients might question, undermining years of effort. If RWE research is published without proper safeguards and is later found to violate scientific integrity, confidentiality, or validation standards, the consequences extend far beyond retracting a single study. Such a failure can undermine confidence in the drug, jeopardize the entire development program, and damage the reputation of the company and even the broader life sciences industry. Trust takes a long time to earn and only a moment to lose.
Where human oversight must remain non-negotiable in AI-enabled RWD programs
Michael Fronstin: Where should human oversight remain non-negotiable in AI-enabled RWD programs, especially as organizations push toward greater automation?
Jessica Santos: AI is a tool; human oversight should always be present in every step. Where it is needed most and how much can be based on risk assessment of individual tasks. There are several risk frameworks we can adapt to, such as the EU AI Act, U.S. NIST AI Risk Management Framework, and OECD AI Principles. In RWD programs, where human oversight is needed most is AI deployment selection, input (for example, only after data desensitizing), prompt generation, and output verification.
Michael Fronstin: I agree. I’d add that human oversight or human in the loop can help ensure agents are deploying desired methods, reviewing variances thereof, and double-checking results to verify hallucinations didn’t occur.
The mindset shift leaders need to build trust while adopting AI
Michael Fronstin: For leaders just starting their AI RWD journey, what is the single most important mindset shift they need to make to innovate responsibly without compromising trust?
Jessica Santos: AI is here already, like it or not. How to adapt to it responsibly is the key. There are many options out there, and choosing the one most suitable for different use cases will set you on the right path.
Michael Fronstin: Jessica, thank you for joining me in this discussion and sharing your expertise. As AI adoption accelerates across healthcare and life sciences, the imperative is no longer whether to engage, but how to do so responsibly. While these industries may have followed technology, finance, and e-commerce into the AI era, the stakes are uniquely higher. AI-driven insights can directly influence patient outcomes, regulatory decisions, and access to care. Embedding strong, transparent guardrails is therefore not optional; it is foundational to trust. When implemented thoughtfully, responsible AI has the power not only to prevent harm, but to meaningfully accelerate evidence generation, advance treatments, and improve quality of life across the healthcare ecosystem.
