Dr. Kellie Owens: Ethical and Trustworthy AI in Health Care, Part 2

Dr. Kellie Owens, is an assistant professor in the section of medical ethics at NYU Grossman School of Medicine. She's a sociologist and empirical bioethicist studying how digital technologies are woven into the fabric of healthcare and society and how governance structures can better support those affected by technological change. She also leads the responsible AI review process for the Division of Applied AI Technologies and with the IRB.

PART 2

In medicine, we do know that examples of bias and harm from that bias have happened. The most famous case was brought to our attention from Ziad Obermeyer and his colleagues in Science in 2019. It was examining racial discrimination resulting from a commercial algorithm that was widely used by health systems to identify patients with complex health needs that might need extra support. Instead, we found that the algorithm was prioritizing white patients over black patients and was reducing the number of black patients who should have been enrolled in those care management programs by more than half. In some ways, there are pretty easy ways to fix these kinds of biases and problems if we're attuned to them. So once we see them, we can usually fix them. In this case, training the model using active chronic conditions rather than total costs would have significantly reduced bias.  The point is that understanding the social context in which an algorithm is developed and is operating in is really important. Otherwise, it can be pretty easy to make mistakes like this if you're not paying attention to disparate impacts in the ways that our choices in algorithm development can have impacts on fairness and outcomes for different groups. Another major ethics consideration touched upon in her presentation is mostly about data privacy.