Health AI Forum, Part 1: Present and Future of Health AI

Episode Topic: Present and Future of Health AI

Explore the future of Health AI, an exceptional product of human genius with immense potential for good—and for harm. Navigate the shift from predictive to generative AI, tackling the urgent need for ethical, human-centered governance to ensure technology serves humanity, not the other way around.

Featured Speakers:

• Fr. Thomas Joseph White, O.P., Pontifical University of St. Thomas (Angelicum)
• Michele M. Martin '88, '90 MA, University of Notre Dame
• Michael Pencina, Duke University

Read this episode's recap over on the University of Notre Dame's open online learning community platform, ThinkND: https://go.nd.edu/891a50.

This podcast is a part of the ThinkND Series titled Health AI Forum. 

Thanks for listening! The ThinkND Podcast is brought to you by ThinkND, the University of Notre Dame's online learning community. We connect you with videos, podcasts, articles, courses, and other resources to inspire minds and spark conversations on topics that matter to you — everything from faith and politics, to science, technology, and your career.

• Learn more about ThinkND and register for upcoming live events at think.nd.edu.
• Join our LinkedIn community for updates, episode clips, and more.

Transcript

Welcome and Introduction

1 0:06

thank you all for, for being here. Um, I am Michelle Martin. I'm the Executive director of Lucy Family Institute for Data and Society. And, this is our fifth AI forum. So, I know, so while the RISE concept and conference, it's our first, this is our fifth, and, um, I'm thrilled to see familiar faces. and sincerely thank you for being here and for being a part of this. how many of you are in health careers right now? Okay. And how many of you do AI or data type of work within your health career? Yeah, so I, I bring that up because, um, when I reflect back to our very first health. Health conference or health, uh, forum. We, it was health equity, health data equity, forum. And it was really focused on how do we include the communities that are so marginalized for both access and outcomes. We then started to pivot into Health Data Forum because we realized that underneath all of the problems that are systemic in the healthcare system, data are, are important to be able to help us to identify where the highest and best use for dollars for talent for impact could be. And now we've even morphed further into a health a, a health AI forum. So I just love that we've taken a, we've taken ourselves on a journey along with you and with your guidance. For those who are new to this, uh, health AI forum, what we have seen in the past, and you'll see several people who have been here over the fact, uh, over the past five, is we find that we, I ideate around concepts, not only problem statements, but also solution ideas. And then what I love is that we end up creating partnerships across lines. So within the room here, we've got payers, providers, we've got pharmaceuticals, we've got government affairs, we've got chief data officers, diversity leads, we've got community leads. we have local, providers and payers. We have regional, we have global. So we are a very nice mix. And each of you has been selected specifically because of. Who you represent, who you are personally, and your investment in these particular areas. So I, um, I'm excited to see where we end up by the time we, uh, conclude tomorrow, because I bet you anything we're gonna ideate, invent, and maybe come up with some new partnerships to go do some of these things. the things that we have done in the past have been very results oriented, driven by the concepts that we come, that come up in the, in this, uh, forum. So I thank you all for being here. I'm gonna give you a little bit of a preview of what we're up to for this forum. and by the way, again, you can choose your own adventure. If you wanna go to some of the other things that are happening in broader, in the broader eyes, uh, context, please do. it just means you'll get fomo because you won't be here, but you know, it's up to you. Um. After this opening when, um, my colleague Fang Lu is gonna talk about the health ex, uh, data exploration and analytics lab, which is a reconstitution of our health AI lab. we will then have a presentation by Dr. Michael Panina, from Duke Medical, who's gonna talk about how he's been using AI in the future, the future of health ai. We're gonna have a round table, uh, where you'll all get to introduce yourselves.'cause I know this is a big room and it's good to know who's in the room. So you can find them for lunch. For lunch. We'll be back over at the Morris in across the street, which is the hotel across the street in the ballroom. But we have four tables set aside for the health people. If you wanna continue to have dialogues around specific topics, um, that may be resonant or relevant. Um, and our researchers will be there and they're taking copious amounts of notes to try to figure out what kind of research should we be doing around this, right? and that's usually where some of the partnerships and the friendships really begin, is in those conversations. Then you'll come back for a broader keynote with Juan Gilbert, and then we'll come back into this room again, health oriented, and we'll have a panel that we've titled Every Person, every Mile, which is about data in rural communities. Our primary topic for this health AI forum is rural health, because it really con, it con it conveys, and it connotes a lot of the problems in America with both access and outcomes. And it also translates very broadly into global health because we have exactly the same kinds of problems just accelerated and exacerbated further. and then, uh, then we're gonna have an hon and I mentioned this a bit in the plenary session. this will be hosted and, and run by 1842 and Alloy Partners, which is a venture fund. And the folks are in the back of the room here, and they're gonna help you to ideate and innovate ideas around rural health problems. And it's will span everything from senior mental health to ideate, on how can we solve some of the problems of Medicare and Medicaid in specific, states where, it, access to, providers is even, is even more accelerated and exacerbated. We think technology can probably have some solutions and they're gonna help lead you through some of those, um, some of those that ideation. And then I'm, uh, we'll have another keynote in the plenary session. And then all of the health students, both local and global. Um, there are 16 of them, who have come from all over to present posters, and I think they would love to engage with you all as health leaders to test their ideas, to come up with new ideas perhaps, but let them shine. Um, let them show, uh, about all the research that they're doing. I, I find it incredibly gratifying. So that's what we're up to today. Tomorrow, I'll give you the overview of tomorrow. Um, it's panels and plenaries as well, but all very health oriented. any questions for me? All right, well, again, sincere thanks to all of you for being here. to Johnson Johnson for being a sponsor of this AI health AI forum, um, and to Accenture for, for actually starting this, uh, back in, uh, 2022. with that, I'm gonna turn it over to Fang Lu. she is, a, she's a, a full, full professor, who has done a lot of work in, the broader, data and statistics around healthcare. for those of you who were working on, uh, black maternal health work with the, with Accenture, with Care Source Kroger, um, for the past couple years, she was in the background doing a lot of the, um, the decision making on how to convey, um, and create the actual longitudinal study. we were absolutely thrilled that she said yes. To leading this new, this new, lab, and it's health data exploration and analytics lab, better known as Heal, HEAL. I love that. So with that fang, over to you over. Thank you.

Keynote: The Future of Health AI

2 8:06

Yeah, that's mine. This is it. Okay, great. Okay. Thank you so much. Alright. Uh, first good morning and welcome. Uh, so thanks Michelle for, uh, opening the session and, and give you overview of what all the sessions and events that we have today and tomorrow. So, uh, actually some of'em, the slides are overlap with what Michelle just said, so I'm gonna skip some of the slides. It's actually not a lot, just eight slides, so a few minutes. Okay. So it's about the lab and, and the sessions, for this conference. Uh, so I'm Fang Liu. Uh, I'm a full professor actually in A CMS, which stands for Applied on Computational Mathematics Statistics. Uh, so I have a few colleagues and, and Tiffany is here. I see my student who is now a postdoc at, at Lucy. Uh, so I got here 2014. Uh, so I'm a statistician. I'm sorry, 2011. Why is that? 2014? Yeah, I got here 2011. I'm a statistician by trainee. Uh, so, uh, anyway, so as Michelle me, uh, I became the director of this lab. Uh. Since July. So it's, it has only been three months. I'm still trying to figure out, okay, so I'm learning about Lucy, and Lucy is learning about me. all So with that, um, alright, so this is the, uh, the vision and our mission for this lab. Uh, so I, I, I believe everybody agreed that, uh, we would envision a world where house burdens are significantly reduced at every level. And, uh, we foster, e ecosystem rooted in collaboration and innovation and open science. And in this lab we seek to advance the common good through data driven insights and AI power solution. And what we do, uh, so hu actually collaborates, uh, with, uh, people from different domains. We collaborate with researchers from academia. Uh, we have business and industry partners, and we also work with community partners, uh, locally, uh, to leverage diverse data sources and, sort of, cutting edge AI techniques to drive impactful and translational research. And hopefully that translate into practical implementation and also, uh, help with, uh, policy making, uh, in healthcare. So I'm gonna give you a concrete example that's done by our, one of our lab members. Uh, where's Hanah? Oh, you are there. So yeah, so this, this work actually was, uh, was completed. It's actually still going on, uh, uh, like in phase two, but it was the phase one, uh, sort of stage of this project was completed before I joined the lab. So I, I do not take any credit, for this work. so this is, uh, as I mentioned, this is done by our lab members, uh, Dr. uh, an Angelica Garcia Martinez. I'm not, I'm, hopefully, I'm saying this correctly, and this is obviously in Spanish. I'm not trying to, to pronounce, but I, I, I, I asked, asked AI yesterday, and it says, it basically means Hello Connect, which is a wonderful name, for this project. so, correct me if I say something wrong. And Angelica, so this is sort of, uh, a collaboration, uh, between Lucy. I believe Lucy actually is leading the effort, uh, with, uh, hospital. children's Hospital and in, in Mexico, and collaborating with, the National Institute of Public Health, uh, in Mexico, uh, to improve the pediatric care, cancer care for low and, and middle income communities in, in Mexico. And it's the, the, the, the team develop a really cool app. And the, the app actually contains sort of like two components. One is used by, Care, uh, by the health care providers, which is sort of a hospital, uh, web component. And also there's another component, uh, used by the families and caregivers. And, uh, those, this app is used to track clinical indicators and treatment plans and, and, try to understand the sort of determinants of healthcare and to monitor adverse events, uh, during the cancer treatment. uh, obviously you are sort of doing a real time data analysis and tracking information and, and security and, privacy and the computation is very important. And they, they build this into, into the app, uh, within design the project. Uh, this is really a cool, uh, as I mentioned, really cool project. And if you wanna know more, uh, feel free to talk with Hanah, after the event. And there's, they also published a paper, so I have a link in there. Uh, uh, go check on Google, uh, scholar and you will find the paper. so I, as this, this slide basically sort of overlap with, uh, what Michelle just mentioned. Uh, we have three pillars, uh, in this health AI forum. Uh, the first one is rural health, and there's a panel, and there's a keynote, uh, speech tomorrow morning. And the second pillar is mental health. and in this particular, uh, forum this year, we're gonna focus on mental health in older adults. And there's a panel tomorrow, uh, morning. And the last one is the Future of AI in Healthcare. And we have, uh, keynote speech, uh, coming up, right after this. Uh, so, and there's also a breakout session. Uh, talk about AI innovation, this afternoon. And, uh, also we we're gonna conclude this health AI forum tomorrow at three 30 by another, uh, wonderful, keynote speech. Uh, uh, tomorrow afternoon around three 30. so. And as, as Michelle mentioned, we have a round table discussion, uh, starting, today at 10 40. And it's gonna continue through lunch. So we're gonna continue talking during lunch. And today we're gonna have four tables. And tomorrow we'll have three tables. Okay? So each table will be led by, uh, sort of a researcher, and, uh, hopefully to guide the discussion. And when you listen to the keynote speech, and we use sort of three sort of pillars here, think about the questions that we could address, uh, using AI and data. and as hopefully that's, that's can, uh, sort of, foster a really, a productive, conversation, during the discussion. All right, so I know why we're, uh, interested in rural house. There are so many topics. and I believe everybody will agree that health is not just about urban hospitals or cutting edge technology in Silicon Valley. Uh, it is a daily journey that, uh, of everybody, uh, in every community, in rural community, often underrepresented and uh, in data, inhouse data and the technology advances, they're slower in adopting, uh, those technology due to virtual reasons. And there's a big gap, right? So if we don't engage them now and that the gap gonna become wider and wider. Uh, so hopefully we can use AI to help solve some of those, gaps and maybe, uh, rather than reinforcing them, right? So it's a sort of a double edged sword, and if we keep using ai, but like, but forgetting rural health and, uh, and this gap gonna become wider, wider, uh, but we try to use AI in the right way. rather than enforcing them, uh, we try to bridging, uh, those gaps. Uh, so you have heard there's like vague, beautiful bail act, by the government. And, and, uh, among this act, there's a$50 billion funding for rural house transformation program. Indiana actually is in the process of applying to this program. I believe there are, uh, several, uh, members maybe here from the Indiana Working Group on Rural House, the transformation program. actually Lucy just put together initiative, on rural house using data and ai, uh, to help with the transformation. And we put in the initiative on Monday. Uh, and if you are serving in that rural house, Indiana rural house, uh, working group, uh, feel free, uh, talk to me and hopefully we can connect, uh, after the, after, after this, uh, this session. All right. So, and then the other ones I mentioned is mental health. In older adults, mental health is a big problem, uh, occur, occur across like different demographic, sort of groups. And in this particular forum we focus on older adults. And, uh, so there's a reason, right? So it's aging research nowadays. It is also very, popular, at least among researchers because the baby boomers, they're sort of aging. And it actually, it's, it is not just the US problem, it's, it is an international problem. Uh, I got this statistic from WHO and which states that by 2030, by 2031 in. People in the world will be aged six years old, uh, six years or older. And so it's, it is a huge problem that the, the world has to deal with. just by looking at the paper, you can feel the loneliness. so loneliness and social isolation are key factors for mental health, uh, especially in later life and older adults, they are often overlooked. Right? So, and, and especially in rural area because of transferred. Transportation hurdles and isolation and the limited access to, uh, healthcare specialist. and we, we hope that technology and AI enabled tools can help to close the gaps, uh, by using some of the advanced, uh, sort of, sort of telehealth, uh, related service such as voice assistance, maybe chat bots or remote monitoring. and of course, right. So, and the mental health is, is also kind of a very sensitive topic. And, uh, when we sort of design those tools and we wanna make sure that they're accessible, to, to the older adults and making sure, uh, their privacy and is preserved and hopefully they will trust those AI enabled us, uh, and the tools and will and start to use them. so the last, this is the last slide. I wanna close the remark we saying that AI inhouse is not, is about humanity. Uh, it is not about machines. It's about us. It's about people. every site, every algorithm should improve somebody's journey anywhere. Uh, so I love movies. If you are a movie lover, you know, instantly, this is from et the classic poster from et. And this is, kind of a human hand touching ets hand, right? So it's, they are now treating, uh, et as enemy. they are friends. Uh, we can connect. The same thing goes to AI and the human right. So when human hands touching the digital hand, uh, hopefully we become friends. Friends we work together. don't be afraid that they're gonna be a replacement of a human. Uh, hopefully, uh, we work together as a touch of reimagination. Okay? Uh, so with that, I'm gonna close the session and welcome to the House AI Forum. Okay. All right. Anybody want to take a break? No. All right. So if No, uh, we're gonna go straight with the keynote, uh, speech today, by Dr. Uh, Michael Pela. And, uh, so Dr. Michael Panina is a professor of bio statistic information at Duke University School of Medicine. Uh, so when we invited Dr. Panina, he, he was still at Duke. He's still at Duke. but, uh, uh, we just learned last week he became the, uh, chief ai, scientist at, at, uh, at U United Health Group, and effective October 1st. Congratulations. That's a big role. and we, uh, Dr. Panina is internationally, uh, renowned statistician and data scientist, and he has published over 400 peer review journals. I just checked the Google Scholar yesterday and he has gathered over 157,000 citations for all. His publication is a huge, huge number. he actually being rated and recognized as the highly citated researchers in, uh, clinical medicine. from 2014, to today and, and also highly cited researchers, uh, in social science, uh, since 2014. Uh, so as I mentioned, his international recognized, and his authority, in evaluation of AI algorithms. And his work has been, uh, used to guide best practices for application of clinical decision support tools in house delivery. He also co-founded and co-leads the national correlation, for house ai, uh, to increase trustworthiness of ai, uh, which is a very important nowadays for adopting AI in real life. Uh, he received a PhD from, from Boston University in, in 2003, uh, in mathematics and statistics. As a statistician, I'm trained as a statistician. I'm so proud of statistic being take, uh, taking such a big lead role, inhouse ai. So with that, uh, I welcome, uh, Dr. Panina to deliver the keynote speech.

Understanding AI Consequences: Privacy, Security, and Fairness

The Need for Evidence in AI Applications

Audience Q&A: Ethics, Bias, and Practical Challenges

3 20:52

Thank you so much, uh, Dr. Lu for the introduction and Dr. Martin for inviting me here. Really delighted, to be here with you and talk a little bit about the future of health ai and, I guess, uh, I will talk about the future or the direction I'd like it to go. I can go, cannot guarantee that it will go in this direction, but this is our, joint task to work on it. So, um, since we're at Notre Dame, I not start with, uh, polio. I found it remarkable that the name of a papal name, is driven to a degree by artificial intelligence, right? This is. Telling us, I think the importance of the technological revolution that we're facing, right? In his first remarks to the College of the Cardinals, he talked about the new revolution that artificial intelligence, uh, is driving and has spoken on AI multiple times. And I think this is an important quote here that grounds us in, in, what we're going to be talking about, undoubtedly an exceptional product of human genius, right? So I think this is very important. So AI has the potential to be good and to make our lives better, but there is, uh, uh, there is the, but it's a tool and how it's going to be used. It depends on those, the intentions of those who will be using it, right? Which puts the opportunity but also the obligation on us. And I will say today. We still can reign in the AI revolution, in the direction we humans can control. I will say, I'm not sure I'll be able to same, to say the same thing, in a number of years and the promise of artificial intelligence, where I see this technological revolution different than the revolutions of the past. The human intellect controlled the development of what's happening, right? Whether we talk about, uh, electricity, whether we talk, about different improvements, uh, right in, in, in production, so on. I think internet and social media, right? The most previous technological revolution has gotten out of hand a little bit, but we still know how it operates. AI has the potential of to be self-propelling. Right. We can talk about AI agents creating improved versions of itself, and it keeps going and going to the levels that I remember from my mathematics education, right? Thinking about dimensions that exceed human comprehension. Well, AI is able to do that. So we are at a state where the technology is more capable, and that's a very narrow definition of what capable means, right? Not to detract anything from the humanity of humans, but more capable in some mathematical, if you will, components of human brain than humans themselves. And it's amazing on the one hand and extremely scary on the other. Okay, let's bring it down a little bit. bring it home. So types of AI used in healthcare. So, predictive is what's been around for a fairly long time, right? I started my career in the Framingham heart study developing risk prediction calculators that the American Heart Association is, uh, still using to guide statin treatments and, and cardiometabolic health. And so I like to say I've been doing baby AI developing predictive algorithms. What's the, your risk of heart attack in the next 10 years, right? With, uh, what now would be considered primitive methods, by the way, a little, side, no AI algorithm with the same data can do any better than the a algorithms from 20 years ago, right? So it's not that AI is, is going to to change everything. Now, if we expand the data, right, and I think then we can talk about, uh, improvement, but the predictive task is okay, we want to know what's your risk of a heart attack in the next 10 years? What is the probability. That on a mammogram, which you're seeing is a cancerous, change that's being detected, right? So both, both prognostic and diagnostic. But the bottom line is there is an adverse event that exists or will be developed in the future, and we use the technology to predict it. So I spend a lot of years, right, starting with my doctoral dissertation on evaluation of performance of algorithms. So not as much development, but I was always interested, are they doing what they're supposed to be doing? And, I've done a lot of work simulations, publications, and so on. So when AI came along, I looked at it from the perspective of predictive ai and I will say by, by summer of 2022, I was very comfortable that I got it. I know how to evaluate ai. It doesn't matter that it's fancy machine learning, deep learning or whatever you, conversion neural network, you name it. The methods for predictive diagnostic applications of evaluation are roughly the same, right? People can throw in additional if few metrics, right? There's a UC and you have F1 and different things. But in general, the approach has been the same. Well, that comfort didn't last long because in the fall of 2022, Chad, GPT came along and basically turned the science of evaluation upside down in many ways for generative applications, right? So you probably remember, look, my experience has been Thanksgiving dinner and, my brother-in-law taking out his iPhone saying, oh, look at this cool app. We can write poems for you, and doing different things, right? And that was the, that was the beginning of, AI completely reshaping. Our thinking of what's possible.'cause now not only it can be used for predictive tasks, but it can generate, can generate a poem if you are into that sort of thing. Uh, but more importantly it can generate, yeah, I'm not sure poems generated by AI how much value they have, but that's, that's my personal opinion. but it can do things that are very useful. I think the keynote session this morning has been a phenomenal example, right, of showing the application ai, including generative AI to really improve business processes and healthcare business processes are broken, right? And so this is an opportunity and that transition us to, to this slide. Okay. So where can AI be applied in healthcare? So, PWE from Microsoft Research, uh, gave a talk shortly after. Chad, GPT was unveil, unveiled and he argued that really basic research is the area where he sees the most transformational potential, long term, right? So that's the protein modeling, that's discovery of new treatments and basically changing the science of discovery with artificial, intelligence. That was not what I was thinking, right? So I'm very much in the kind of applied clinical domain and the AI applications I've seen by 2022 at that time we're predominantly in the diagnostic and prognostic, right? Let's discover, predict new disease or discover the, the existing disease. So that was really interesting and you see more and more of it than that. Some venture capital conferences and people are taking bets of what percentage of new treatments would not be there in 20 years. If not for the application of artificial intelligence, right? Then the bets are going 50, 60, 70, 80%, right? So really, really high. So very important domain. what we have seen kind of pre 2020 pre pandemic pre-chat GPT era was clinical applications, right? There was the, Watson, IBM Watson that basically tried to promise doctors that it's going to be smarter than them and it will be useful. I think the technology was decent, the presentation not so much'cause doctors about, we don't need it. So, that, that has not been a, a success story, at the time. And then what we've seen in our health system at Duke, a lot of developers, either internal creative faculty building new algorithms or external. Right. And some in partnerships, a lot of companies bombarding our health system and saying, we have this great AI tool, it's going to predict this. Or it's, uh, diagnose this and it's going to make patient lives so much better. Patient will be healthier, saves you tons of money, and basically, uh, do miracles, for your health system. And we were not ready as a health system to validate these claims. What we were good at was data privacy and security. And so that became the criterion of assessing the new technologies coming in. Right. Well, and as you can imagine, you probably heard or, or or read about the nurses at Kaiser Permanente going on strike, I think a year or two ago because of ai. We had a mini version of that at Duke in 2018 or 2019 pre pandemic where the nursing staff told us or bringing this ai, I don't know what it's doing, I don't know what it's doing, and it doesn't align with my clinical practice. So I don't wanna use it. Right? And so that started sending the message that we need to think about it in a different, different framing. Now the pandemic came, I'll get back to how we solved the governance issue. But pandemic came and GPT came and the focus of AI application switched from clinical, which are still, still being developed. Right? The most recent, I think paper from the FDA says that they've approved about a thousand different AI based technologies move to the operational, right? A few reasons. Generative AI lends itself to operational applications. Of artificial technologies in healthcare, the risk is lower. You don't have to go to the FDA or, I guess developers think they don't have to. And the FDA is not sure if they do. There is a large regulatory green zone, uh, gray zone, not green zone, uh, uh, in existence, but a lot of waste and opportunity to improve the processes. So a lot of focus switched into the development of operational AI technologies. And I'll give some examples, right? So, but that's the kind of universe, as I see it. So the potential is incredible and it touches so many domains, right, of, uh, of healthcare from, randomized clinical trials and evidence generation in research. identification of future risk, right? So the clinical, assisting in in diagnosis, elevating clinician burden, I'll get back to it, but the most, I think successful widely deployed technology is the ambient scribe, right? So it's the technology that listens to the patient provider conversation and writes the clinical note. And we've, uh, rolled it out at Duke. I've been, uh, involved in the process. I was also involved as a. As a parent of, of my children seeing a pediatrician, right? So it's like, my wife takes a child to the pediatrician and then, before they're home, I can read the note and see what's been happening. And it's, uh, uh, it's incredible. The doctors love it. The patients love it because it brings the human connection back, right? Instead of the doctor looking at the computer and typing and double multitasking or doublet tasking, they can have a conversation. And the ambient technology is there in the background. efficiency, streamlining, operations, right? A lot of, uh, focus now on, on improving claims processing, patient engagement and satisfaction. Now, you can get AI enabled if you're undergoing a surgery right before and after, you can communicate via text, uh, with an AI technology about preparation. So, a lot of great things, uh, are being built as we speak. and again, I think another quote from, from Paul Leo and taking us, uh, us back to encyclical rum Novarum, right from, over a hundred years ago, it also presents new challenges for defending human dignity, justice, and work. Right? And the question I think Fang, you alluded to is, um, ai, will it take away human jobs, right? So that's the how will it, will it transform it or will it change it, right? That the, somebody, the, the big question, one question medicine is, well, will radiologists be obsolete? And one answer that I like and hope that's the direction is that, well, it'll not replace radiologists, but radiologists who use AI will replace those who don't. Right? That's, I think that's the direction that, uh, that we can embrace, but it gets deeper. Robotics is progressing, right, where we talk about very desirable claims processing, being automated, which means patient gets approval much faster. Well, there are armies of humans, oftentimes nurses, right? Skilled, skilled clinical personnel working on it. What happens to them? Well, the positive is they go back to patient care, what they, wanted to do in the first place. The negative is, well, will there be enough jobs? I think as we saw at the, uh, keynote, um, last night, is we'll have a shortage of clinicians, both doctors and even more nurses. I think there is, there is a fantastic study by Accenture, pointing out to it. So that might be, again, huge positive opportunity. But you can talk about more and more, right? Self-driving cars, self-driving track. that line of work will be replaced. What about the infrastructure along the highways that will be replaced? So that's all happening and it's not clear that it will all be good augmented humans working with it.'cause there is a lot of business push right to, uh, to really, bring in cost savings. Somebody saying the cost of an Optimus Tesla robot is supposed to get down to 20,000 a year. That's very hard to compete in terms of numbers, right? With employing a human being, nobody will be right. A reasonable salary in this country will never go down to, to$20,000. So there are economic challenges in front of us that will affect healthcare. So. Going to principles. right. The, Vatican has been very proactive and invited, actually a, leaders, thought leaders from different religious and ethical traditions, and issued the, this call for AI ethics, right? So, people from all different backgrounds around the world gathered together, in Rome right before the pandemic and work together on this, AI ethics set of principles. And they've, uh, enumerated six of them, transparency, kind of knowing what it's doing. Inclusion right? We're touching on that in a very important way. The participation of humans in the process and not replacing the humans accountability, which is the responsibility of what's produced impartiality. Avoiding bias, reliability, making sure that the thing works, and then security and privacy, right? Related to our confidentiality, data, and so on, in a similar tone. about three years later, the Coalition for Health AI put together its set of principles and the coalition as an organization started by academic, institutions, but very quickly joined by a lot of industry. And now over 3000 members of CHAI in the largest health AI organization, in the world. Primarily the largest, uh, segment of membership is smaller companies, right? The developers of ai, that, uh, that want to, uh, want to participate. There is, uh, patient organizations, patient advocacy, a lot of non-profits. So the first document CHAI put together is the principles for trustworthy ai. And interestingly. Even though, it has not been aligned, at least to my knowledge, right? Chai and the Rome call, the principles are, are quite similar, which speaks to the, kind of, true consensus that's, that's emerging. And those are usefulness, right? So that's valid, beneficial, testable, reliable, usable, kind of the, it, it works and it's, uh, it's beneficial and safety, right? Um, so that, that's the two going together, the benefit and the safety of the technology. Then you have accountability and transparency. So the responsibility and that we know what it's doing and why. And explainability, interpretability that is being challenged with GPT, which is more of a black box than even other technologies. But we still need to seek to understand the consequences at the very least, privacy and security and fairness. avoiding bias. So, taking it maybe a step down, right? So when, I was asked, given my expertise and evaluation of performance predictive ai, when generative AI came along and said, so what should we do, in, with generative algorithms? And I spend a lot of time, right, studying metrics for predictive, the AOCs and, and, uh, propose some metrics. And, I could not answer that question, right? We, a predictive task, the event happens or not, or, the diagnosis is there or not is very different than reading a text. And so my response was, well, I think we need to get to very concrete principles for evaluation of artificial intelligence, right? So I kind of asked myself. How do I think about it in that context, learning from the predictive or diagnostic that would be applicable across, and the foundational component is ensuring that AI technology serves the human person. And again, you would say, well, it's obvious. Why, why else will we build it? I don't think it will, it's a given, right. It will not be obvious'cause, profit and other motivations might start dominating and you let AI lose and the agents are going to create more agents and there will be a parallel universe, right? It's, it's, uh, already, avatars exist and the, generation coming up is, uh, is uh, embracing it very quickly. So we need to anchor it in really service the human person. That also means that we as humans need to define or get to define what we're going to apply artificial intelligence to.'cause there are applications of value. Some of those examples that I've given, and there are applications that are not of value, right. Are degrading to, to human dignity and we should not pursue it. Right? So on the very high level initiatives like the ROM call and the, global agreement of not, crossing certain boundaries, I think is necessary in the human versus machine. I think there is a lot of focus for good reasons why individual countries are rushing to the AI development, but without some form of global consensus of the areas we don't want to go, things are very likely to get out of hand. And describe this, what the successful use of AI technologies look like and what are the safety concerns? Very important in healthcare, right? We have not done that as a society with social media, right? If I tell you, trustworthy social media, what comes to mind? Well, probably an oxymoron, right? and so we failed. let's do better with health ai, right? Let's define the consequences we don't want to happen and what good looks like, and build and monitor with that in mind, and then to be able to do it. The fourth principle is we need transparent systems for, for governance, testing, and monitoring, right? what I worry the most right now. Is the AI kind of happening on the fringes that nobody knows about, and it explodes and creates, very, uh, negative adverse consequences. And fringes doesn't mean it needs to be somebody's garage. It could be the health system that's operating at a doctor, build an agent, and the agent does something that's, that's really off. Or even a company built a tool that was doing well, but with data drift changed and from good, started being harmful. I also want to, make sure that we don't. kind of box ourselves into the principles which are kind of focused on, okay, let's make sure we don't do these things right. The kind of the limits of it, the, positive, right? Uh, in a philosophical sense approach is also thinking about virtues and the training and education of new generations of AI developers and users. so principles, right? Kind of are, are creating the guardrails, but, they will not protect us, from, from everything. So. Looking at virtues for, AI systems, right? Where clinical encounter, for example, between a doctor and a patient, it's very hard to tell the doctor, well do this, and this. Don't do this, this, and this.'cause every instance is different, right? So the doctor has said, ai, user trained in virtual or a nurse, right? is, I think gives us a better chance of ending up where we want to than just being said of the, of the dos, and do nots. And the same with development, right? People who are building this technology. Usually when you study math or computer science and whatever, you don't think about ethics, right? You, uh, oh, philosophy. That's, that's for philosophers, right? I dunno. Ancient Greece or, or, uh, times long past. And I think this is changing and it has to change because of how, uh, impactful and powerful the technology is. We need to bring it together and view education in a much more holistic way than just, okay, I'm a quantitative person, or, I'm a humanities person. I think the lines are blurring as we speak. And, this is really important to keep in mind. So we live in a wild west of algorithms, right? We've, we need sheriffs to, uh, reign it in or otherwise who knows where it's going to go. famous paper from 2019 about bias in, in health ai, applications. Right. Kind of, uh, put the, uh, the negative, uh, implication in the forefront. I was actually very upset after that paper, came out about, okay, people are missing basic things. What happened is the algorithm used the proxy, right? Instead of true health status used healthcare utilization, which of course people with limited access not utilizing healthcare, which does not mean that they don't need it, right? And that created huge bias in the algorithm. Basically said that 17% of, uh, black Americans need preventative services. And the reality was 50% needed it. So enormous amount of bias, right? I was. Kind of, this is one of the do nots that I knew from my Framingham and other experience. And we published the paper kind of reminding people there are principles, right? The final outcome correctly, don't use a proxy that does not represent, what you're trying to measure. we also made this, uh, uh, turned out a prophetic statement that given the number of emerging algorithms and application, no single regulatory agency can review them. All right? This paper appeared in, April, 2020. I say it's my best, less, least read paper'cause nobody cared about AI and governance in April of 2020. But this prediction came true in, four years later in the spring of 2024, where the commissioner of the FDA at the time, Rob Kiff, said pretty much exactly the same thing. The FDA cannot do it alone. Don't expect us. It needs to be more of a societal partnership working on the evaluation of this technology. more recent paper with our head of, of cardiology at Duke. We talk about the need for evidence, right? In, drugs and devices. We have good processes, clinical trials, post-market studies, right? The evidence is generated. We don't have that with ai. And again, not all AI needs the same level of evidence.'cause some applications are more on the IT spectrum. Some are very much on the clinical software as a medical device spectrum and some are in between with the gray area. But we definitely need more evidence. We will not develop trust for AI technologies. Without more evidence. And this is a very, very important gap. And I think that's where, academic centers, universities come to play a very, very important role, right? Designing good studies, whether pre-market, post-market, in evaluation of performance and impact of artificial intelligence, technologies. Okay? Risks abound, right? So it's, uh, non actionable, predictions to wrong predictions, technical malfunction, failing to improve, outcomes. So it's might be the tool itself, or a lot of attention is focused, but maybe equally important is how the tool is implemented in the context of the clinical workflow. How are we using the ai, right? You can, you can have a good tool that's poorly implemented. Or not, not embraced, not performing well. And there are examples of going too far, with the application. So two quick examples. I talked about ambient, right? Automated clinical notes. So starting, what's the problem we're trying to solve? Is generally AI a good tool? it appears it is. What are the implications in which the algorithm was trained, right? Is it going to perform well for patients with different, ethnic origins, different accents, right? So do we need to, uh, and how do we catch the errors? Who is accountable for the notes accuracy? We had a fantastic discussion in our AI governance at Duke. What should we ask or require of the clinicians using this tool to make sure that they read the note and sign off of it? Right there is the danger of automation bias or complacency. First notes, like hundreds you will read and be careful when it gets to thousands or hundreds of thousands of notes, right? Uh, as you, as you see the patient, you're like, well, it's doing well. So I can just go, I, I'm tired. I can, uh, maybe see another patient or spend time with my family. So that's, uh, that, that's a real danger and privacy and compliance, making sure that the information doesn't leak. And more scary example, a brain chip, right? We hear about development of, um, kind of augmented human intelligence. And so let's take the poor radiologist, uh, as an example, right? So imagine that the brain chip would be, put into the radiologist, that it would give them the ability to have the AI knowledge from all the radiologists on whom we have data, right? So augmenting their ability. To know that much more or be much more efficient, right fast in how they are, making the decisions. And that is absolutely scary, right? Going through the Transhuman domain, what I worry about, the drive for efficiency and profit is going to, to, uh, proceed, right? And that opens a lot of question. Like what about if, one, one doctor says, I definitely don't want anything to do, but others say, sure, yeah, augment me. I'll, I'll be better. My quality of life will improve. Right? Utilitarian perspective, can be, easily shown here, right? that might make things, better. And then we get into issues. Well, now companies might want to hire the augmented doctors with, brain chips. Over the ones who are not augmented, that, steps into right, our agency and freedom of choice. So very, very scary dimension, right? That, again, let's be mindful of it. Those discussions are, right now, we put them kind of in the science fiction categories, but we need deep ethical frameworks, right? And that's the call for us to, to act now and again. quickly I talked about governance. So this is what we've been doing at Duke, right? Uh, following the, uh, uh, bias paper and our response to it. And also. Being bombarded with AI technology developers not knowing what to do with it. We put together, AI governance, which was probably the, the first one among health systems, uh, nationally out of primary focus on patient safety. We called it algorithm based clinical decision support, A-B-C-D-S. And basically with commitment from Duke Health System leadership, the decision was made that no new algorithm whether AI driven or not, will be applied to Duke patients until it goes through a review, by the, by this, by this board. Um, the process was designed following what the FDA does with medical devices. Right. So four stages, starting with the development that the pre-human phase, the silent or local evaluation effectiveness, which is the pivotal trial, and then general deployment with post-market monitoring. The other important component that we stress this registration of the use cases, right? I think it's foundational for governance, is to know what you have because you cannot manage what you don't know that you have, right? So having an inventory of the AI technologies that are deployed across your organization, in this case a health system is necessary. So we moved into, requiring any tool that's considered for use on the patients has to be, has to be registered, and then we have these checkpoints and monitor the progress, right? Uh, in another paper last year. we postulated, every health system having a registry, and then maybe on a voluntary basis having the registries talk to each other, right? So if we have an ai, good use case or malfunction, we can share with another partner health system that's running the same technology and learning, learning from, from each other. We took it a step farther and actually we, we partner with, a, a Accenture to build a technology called, uh, smart AI Governance Engine, which makes it, uh, makes it possible in an easy way.'cause it was kind of a mess. We tried to use, uh, Excel box email, right? Coordinating the process was difficult. Now we have one elegant platform. Those of you who are familiar with, with, IRBs, it's like an IRB for algorithms, except that. Much nicer than the IRB platforms that I'm familiar with, so operates much more smoothly. We spent two years making sure that it's, it's, uh, it's a pleasant user experience. But also the learning has been that the evaluation we designed five years ago or we're kind of thinking, okay, we can evaluate every algorithm the same nice FD alike process, right? That became impossible. We were too resource constrained and there was too much AI coming along. So we switched into a risk-based evaluation framework, right? So the applicant submits the registration, the governance team performs a risk assessment and decides, okay, is it a higher risk? application. Is it medium or is it lower risk? And that it's the consequences of it as well. Proximity to patient health, right? Are important features. So you could do full review, right? You put your resources at the highest risk technologies and for the lower risk, you might just go straight to registration and do monitoring over time. And the risk level we spend a lot of time discussing, right? It's the direct clinical decision making. the closer to the patient, the more scrutiny you need to deploy. Again, not everything that touches patient is automatically high risk, right? We have examples like, well, sepsis prediction algorithm. That's definitely high risk. We have another one, which is advice for, uh, matters related to lactation, right? For, that's probably not high risk, right? So it's, it's, uh, consequences based. And also there are operational. tools that might be high risk. We're working on a tool that does assignment of operating rooms for surgeons, right, with the power of ai. So on the one hand, well, does it touch the patient directly or not? Well, maybe not directly, but that might delay a procedure for a patient, right? So we need to be, we need to be careful and we need post-market surveillance, so we have not done well. again, as a society, as a nation in post-market surveillance of drugs and devices, we need to do much better. And ai, because of the unpredictability and potential to change in the application requires post-market surveillance even more than drugs and devices, right? I think the my thinking is maybe. There is a way to make the innovators happy and make the users happy that we create a framework and maybe try regulatory sandbox in which we say, okay, we're gonna shorten the path to market for the developer, right? With transparency in exchange for commitment to participation in post-market surveillance and monitoring of the technologies. Because from a perspective of a user health system, I wanna know that the tool is good today. But what I really wanna know is that it's going to remain performing like it's supposed to over time. And we don't have that, today. And again, that can create a metadata registry. I, published a paper with my colleague, cap Colman. recently where we're postulating this national outcome metadata registry that we learned together about the application of AI in healthcare. And every health system does their own governance and collects information. And we might use agent AI to summarize some of the, some of the information, right? So AI again, can be used, uh, as an ally here. So, looking into the future, I think education, virtue based and workforce development, absolutely critical, right? It is a priority. And again, that's where universities who are trying to define, what education means in the time of artificial intelligence, this is the opportunity, right? Not shying away from ai, like we have to, learn and use it, but creating the strong ethical foundation across all users, right? No longer engineering students kind of can forget about, uh, uh, ethics. I think it needs to be. Fully, fully integrated evaluation, generative and agent ai and, what it means, methods that go beyond human evaluation, monitoring and safety surveillance. I mentioned absolutely critical and assessment of value. I think the next few years we need to move from hype to value, as alluded to in the, in the morning session, right? The value cannot be, is not a given and many users, right? the humans who are supposed to use the AI are not sold on the fact that whether it's copilot or any other tool brings value. And we need to explore it and we need to demonstrate it. And value doesn't only, mean money, right? There could be things that are cost saving, but are dehumanizing, right? Or, or creating problems down the line. So we have to be thoughtful, uh, how we think about it all. I'll stop here and take any questions. Thank you.

2 57:53

Thank you so much for the wonderful talk. so any questions from the audience? Oh, right there.

4 58:00

Uh, thank you very much for the fantastic talk. I completely agree with your point on ethics, around principles,

5 58:07

uh, about principles being ineffective in practice. there's been some work in healthcare ethics, which suggests that virtues, uh, the cultivation of virtue is also a deeply ineffective, task. And I feel like there's even more distance if you're creating an AI system between, you know, if it's possible, which I would challenge theologically, that you can cultivate virtue in a useful way. the feed through from that to a system, is I think, quite a contestable, um, path. I just wondered if you were open to reflecting on some other ways of doing this. For example, looking at outcomes, rather than the cultivation of virtue.

3 58:43

So, yeah, I, I think looking at outcomes is absolutely critical, right? So my talk about the monitoring and surveillance, I am, I'm more interested not the technical component of the algorithm, right? So trained as a, statistician, so it was a UC and, and the metrics and the, the older I get more life experience I have, I'm starting to see how little that thing is, right? It matters, but it, it is the larger context. So I think the looking at outcomes makes, makes a lot of sense to inform right? And deeper understanding of the, uh, uh, of the development process, right? So, and the challenge with outcomes, right? So my, the four principles I listed, kind of write down the adverse things and the positive things that, that you're expecting. It's also important that you just don't think about the. Things you would expect the technology to do, but the unexpected. Right? So to give an example, my first five years of my work I spent on, on coding, right? in, in statistical analysis. I was not good at it. I didn't enjoy it, right? I wanted to kind of look at new methods and other things, but I had to do it. then when I started having people do the coding work for me, I felt I was much more knowledgeable, right? In, in, in being able to assist their work and understand what's actually happening. Now AI can replace the coding. The question in my mind is it, is the new generation of people kind of on my path, are they going to miss out because they didn't spend five boring years coding, right? Uh, as a, so that's just one example of the consequence. That's probably, you don't think about it, right? When you, when you're looking at an outcome and say, well, I'm gonna use ai, it will be faster. Code might be more accurate, but what are, what are the downstream application in healthcare? very much, very much, the same. I, I still will argue that principles and virtuous definitely have their space, right? I think looking at outcomes can inform it. I would rather, go see a virtuous doctor then one is not, or have an engineer that had that ethical training. Think about how they're building the ai. then somebody, that, that has not been at the same time where it gets kind of, transhuman is the, trying to think about virtuous systems. So the virtual AI is automatically that, that really worries me, right?'cause that's the, that, that's a bridge too far and I don't think we can achieve it. I think the quote from polio is, is, is really good here. It's a tool and it's the humans who are gonna decide how the tool is gonna be used. Rather than the tool being ethical of its own. Yeah. I, I don't want, I don't want to worship in, in the temple of ai.

2 1:01:37

Any other questions? Oh, okay. Sorry, go ahead. Yeah.

6 1:01:43

Hi. you mentioned, uh, do not use a proxy for what you are trying to measure. Could you speak a little bit more about your approach of identifying mitigating bias?

3 1:01:52

right, so I think the, the example, right, of the, uh, of the proxy that, that we use, that, well, the, the Obermeyer paper has shown, right? It's a known paradigm in, in clinical, clinical research that you measure what you measure and if you have to use a proxy, make sure that the proxy represents to what you want to have, right? So the definition of outcome is absolutely critical, but, Sometimes it's hard with the data that we have, right? So we, do have the, concept of inform, informed presence bias. And that's the idea. If you want to use electronic health records data, which is being used much more, right? And you use hospital data and you want to study people, in their, twenties, thirties, maybe forties, right? Who do you get? Right? So you get very few men with preexisting conditions, and the women you get are pregnant women, right? That's the, and then you develop algorithms and train AI on that. Well, that is not representative for the tasks that, that you want to, apply, right? And there are a lot of examples, right? Pulse oximetry during the pandemic, right? trained on, uh. Why it's not applicable to people of different, uh, skin tone. Um, one has to be really, really careful and people are proposing methods okay. To augment the data and do different things. That worries me. I don't think we can overcome, lack of data quality with mathematics, right? And so the, we have to improve our data collection processes that we have the data to, to develop things. And then we have to be careful how we set it up, right? not going to the proxies that are not informative again, not taking it too far, right? I, I, do not like extremes in general and saying, well, the data is not good enough, so I'm not going to do anything, right?'cause you always ask a question, even if what you're doing is imperfect. Does it improve the current state? Right. That needs to be the comparison again, the example from the bias paper. Well, they could have done it much better. Right. And then the paper has shown how, how things could be improved. So there is still a lot to do, but what I learned from my causal inference colleagues is not every problem can be addressed with the data you have and causal inference methods. And unfortunately we try to force it too often.

7 1:04:35

Thank you so much. This is so interesting. you talked about the concept of data drift and that a tool can work and then over time it might not be as accurate. And this post-market surveillance as a non. data scientist, can you help me understand the speed at which that might happen? if something is working well, is it enough to do monitoring and safety quarterly or something like that just in a very practical way, or will it have to be constant surveillance?

3 1:05:08

That's, it's a fantastic question, that we spent a lot of time discussing. I think the temptation of people working on monitor a lot of companies, very well-meaning and well-intentioned, that they're building platforms for monitoring what we've learned in our Duke experience. That every AI technology is different. And if I'm working on a 10 year risk of cardiovascular disease predictor, I don't need quarterly, annual is probably good enough. Maybe even a few years. Right. Looking, looking at what's happening. The shifts will not be that fast. Now, the shift will happen if your population, you use the algorithm, identify people who need a statin. They take the statin. Right. And their, impact of their lipid profile. Right? This is very different. My favorite Framingham example is the treatment for blood pressure we put to the algorithm. Treatment for blood pressure had a hazardous ratio of 1.5, right? I read literally would tell you that being on blood pressure treatment increases your risk of cardiovascular disease by 50%. Well, that's nonsense. it's a, uh, it's the uh, right causal inference phenomenon. People who are on treatment are already at a higher risk. The treatment is helpful in reducing the risk. But the algorithm on the face of it is showing what it's showing. But you need to spend time, if you're doing a sepsis algorithm right, which is, very, very acute hours, you need to monitor it much more actively than 10 year risk prediction algorithm. And so the problem why AI monitoring is so challenging is every solution you need to have a plan targeted for that solution. But that takes time and effort, right? in, in putting it in production. But there is no other way. If we don't spend the time, these things will run. And if we try to put them in, in kind of, generalist boxes, uh, we'll have, uh, adverse consequences.

2 1:07:10

Great. So we take one more question, then we can Okay, go ahead. I,

8 1:07:12

I just wanted to build on Robin's question'cause the automated notes. And you referenced how, and I've seen it in my own life now on Google, I rely on the AI generated answer way more than I would've in the past. And my daughter-in-law is a physical therapist and she spends significant time doing notes at home and she has a baby. So she's gonna pick the good of spending time with the baby if she gains confidence that the AI's doing a good job on the notes. And I'm just wondering practically. How do you stop that? Like it, it seems like it'll be almost automatic that people will do that. And I, and I know there's risk factors and legal will say, oh my God, it has to be gone over every single day to make sure it's accurate. But how practically can you stop people from just getting a little lazier and just relying on it?

3 1:08:09

Right. So we had, so I, I think there is, there is a lot of depth and discussion, right? So the discussion we had at Duke on the topic, so one was, one view was let's have the, clinicians trained and sign off that they will read every note and have them re-certify every year. We actually abandoned that.'cause there is, that's an added burden they have to sign off on so many things. We said, let's develop training, right, that they have to undergo reminding them with constant reminder. So that's. That's one thing. The other thing, the monitoring, right? So you have it run in the background. I'm not opposed to using AI for some of the monitoring of the accuracy. Right. Is there an alignment that it, okay, it's because there are examples, like I've seen some Microsoft presentations about their performance tools, right? That there is the risk that you assign, the medication from a different class, right? That's where it gets really, really scary. And then you talk about agents that can automatically send the prescription to the pharmacy, right? That's, those are the things we'll need a lot of focus. And I agree with you, I think I take a lot of comfort from knowing that the clinician reads the note, but there are already examples, right? So the, our health system example from anecdotal from a colleague, right? She went from an advanced imaging procedure. And, AI read the image and said, you have a major neurogenerative disease. And actually she had that in, in, in the family and she got completely scared, right? Dr. Started thinking about writing a will and, and all that. it was signed off by a junior, physician who just followed what the AI said. Now, the AI was not wrong per se, in the sense that it was calibrated to be very sensitive, right, to flag everything. But there was something lost in translation between the physician who trusted that, right? And she went to a specialist who said nothing to worry about. Like, we see this all the time, right? So here is a story of kind of calibrating design and training the people how to use the AI rather than blindly trusting. And we need stories like that in the education materials to say, this can go wrong. Right. You cannot be always trusting it. It's your responsibility, to make it happen. I hope in your example, the burden reduction from having the scribe assistant is worth it. spending a little bit of, little bit of time reading the note, right. So hopefully the, the, the overall value is such that, that we can get there. But again, as as alluded to in the, in the, keynote know session this morning, the training the human in the use of AI is much more difficult than training the, the, the machine.

1:11:06

first of all, let's thank.