The pulse oximeter problem: a trusted medical device comes under the spotlight | Michael Sjoding

MS: 00:00

If you recall back at beginning of the COVID-nineteen pandemic, there was a lot of concern about COVID. Am I getting sick with COVID? People were buying pulse oximeters from their local drugstore, right, and taking them home. So there was just this heightened awareness about pulse oximeters and their importance.

TS: 00:18

This is Dr. Michael Sjoding, Associate Professor of Medicine at the University of Michigan.

MS: 00:23

Around that time, we had the Black Lives Matter movement in the United States. So I think people were just primed to sort of see our study, understand its implications in a way that past studies just weren't able to do, I guess.

TS: 00:39

He's talking about a study he led that shed new light on disparities in the performance of one of the most common medical devices, the pulse oximeter, which doctors use to measure blood oxygen levels. Dr. Sjoding and his colleagues conducted the study in twenty-twenty in the early months of the COVID pandemic. Their findings - that pulse oximeters were less accurate in Black patients than in White patients - were later published in the New England Journal of Medicine.

Today on Made for Us, we talk about the impact that study has had and what it says about the design of medical devices. Dr. Sjoding also reflects on how it's changed how he approaches patient care. In a future episode, we'll meet Valencia Koomson, an engineering professor at Tufts University, who's developing a pulse oximeter that works across all skin tones, and she'll unpack what makes it such a hard problem to solve. This is Made For Us, the show where we explore how intentional design can help create a world that works better for everyone. I'm your host, Tosin Sulaiman.

Here's my conversation with Dr. Michael Sjoding.

MS: 01:40

My name is Michael Sjoding. I am a pulmonary and critical care physician at the University of Michigan, so I'm a lung doctor. So I take care of patients with lung disease, both in the hospital setting and in the clinic.

TS: 01:58

Right. So let's start at the beginning. When did you know that you wanted to be a lung doctor?

MS: 02:04

When I was in medical school, particularly in medical school, I really thought the lung was a very interesting organ because it's so essential to health. It brings in oxygen. It removes carbon dioxide. There's just a lot of really cool physiology and physiologic principles at play. so thinking about being a lung doctor and the disease of the lungs and how those things interplay, to sort of reduce people's health, I guess, was what I was really interested in. So I really decided probably towards the end of medical school that I wanted to be a lung doctor.

TS: 02:41

And presumably you're happy with your decision?

MS: 02:44

Yeah, absolutely. It's a great balance of both caring for patients in the hospital with lung diseases and in the clinic. And another thing I will mention about my career is that I spend a substantial amount of my career also doing research related to lung disease as well. So it's just been a great balance in a really enjoyable career. And the other thing I will say is there's been a lot of progress, even since I've started as a physician. We've seen a lot of progress in sort of new treatments for patients with lung disease. So it's been an exciting time to be a lung doctor.

TS: 03:20

And so in twenty twenty, you made a discovery about pulse oximeters and that's what we're going to be talking about today. And I found it interesting that you called it serendipitous. Tell us what happened.

MS: 03:32

Yeah, so I'm sure everyone remembers, but twenty twenty was the very beginning of the COVID nineteen pandemic. And so I practice in Ann Arbor, Michigan, which is fairly close to Detroit, Michigan, which is one of the larger cities in the US. And Detroit happened to be one of the cities that kind of had a large influx of patients with COVID early. And pretty quickly, Detroit and Detroit area hospitals were really overrun. so University of Michigan Ann Arbor took in a lot of patients from those hospitals. So pretty quickly on in COVID, we had basically created a unit with very sick patients with COVID-nineteen that we were caring for. And as lung doctors, we were all sort of taking turns doing a week in the COVID unit.

And for those who don't know, Detroit is a very diverse city with a large minority population and a patient mix that's sort of different than the typical patients that we would care for at University of Michigan. So during COVID, during that first influx of patients, the number of patients who were black or African-American was really a lot higher than we typically saw. And then the other piece of this is that to care for patients with COVID who are very sick, we were paying very close attention to their oxygen levels, right? Because detriments in oxygen was really sort of how patients with COVID did poorly.

MS: 05:08

So we were monitoring them very closely, especially in this advanced unit. And to do that, we were measuring both pulse oximeter values of their oxygen levels, as well as arterial blood gases to confirm their oxygen levels and their carbon dioxide levels in their blood. So really we had, just to summarize, a population of patients who were very sick with lung disease, a population where there was a larger proportion who are minorities, like black, than we had in the past and a lot of measurements of their oxygen, both from the pulse oximeter and the arterial blood gas. So all those things together sort of was a recipe for allowing us to examine the accuracy of pulse oximeter in a way that we hadn't had before.

TS: 06:02

Yeah, that's really interesting. So before we dive into what you discovered, help us understand what makes pulse oximeter so essential. So when you're treating someone in the hospital, what does the device actually tell you that you can't necessarily see with your eyes?

MS: 06:19

Yeah, so pulse oximeters are basically trying to measure how much oxygen is in the blood. And, you know, that level of oxygen is critical for our ability to function, to live, essentially. And a pulse oximeter specifically tells an oxygen saturation, how much of the blood is saturated with oxygen. And we typically would expect an oxygen saturation of like ninety five percent or higher.

And we use that measurement, the pulse oximeter oxygen level, all the time while we're caring for patients. It's really, in some respects, a vital sign in the same way that we pay attention to someone's blood pressure, someone's heart rate. In the hospital, we're also paying attention to their oxygen level. And it's very difficult to know what a patient's oxygen level is without that measurement.

MS: 07:14

You know, you can kind of see a patient maybe with blue lips. Maybe they can act, they act a little funny, different, off if they have low oxygen, but it's really hard to know that for sure without a pulse oximeter. And so the pulse oximeter is this really remarkable technology, frankly, that allows us to get a sense of what a patient's oxygen level is. And we use the measurement to make a lot of medical decisions, particularly to understand how sick a patient is. For example, do they need to be in the hospital? How severe is their lung disease? We make all these types of decisions based in part on what their level of oxygen is.

TS: 07:54

And you also mentioned arterial blood gas. So it's not the only way to measure blood oxygen levels, but it's the preferred method.

MS: 08:02

That's right. So a pulse oximeter typically is a little device that clips right onto your finger. And that device is basically shooting wavelengths of light through the fingertip. And based on how much of those wavelengths of light makes it through, the pulse oximeter is able to sort of make an estimate of what the oxygen level is in the blood. A much more invasive method for getting oxygen levels is to do a specific type of blood draw where the blood is taken directly from an artery, typically from an artery in the wrist, and then take that blood and measure it in a machine. So that's a very invasive way of testing someone's oxygen level, whereas the pulse extensibility is so simple, it's very non-invasive. So that is much preferred. So we really try not to have to do arterial oxygen levels in case we feel it is very necessary.

TS: 08:58

And by invasive, you mean painful.

MS: 09:01

Very painful, yes.

TS: 09:04

You said that if you're looking at a reading that's ninety five percent or higher, that's good. At what point do you start to get concerned?

MS: 09:11

Yeah, so I think in general, a reading below ninety five starts to make you concerned. And then in particular, readings below ninety are serious. So a normal healthy person with normal lungs, we would expect their oxygen level to be above ninety five all the time. And then the other thing I will tell you is the medical practice is that when a patient's oxygen is below ninety, that's when we're really needing to provide them with supplemental oxygen. So we tend to think that when a patient's oxygen is below ninety, that's when serious harms can occur. And that's when a patient would get treated with supplemental oxygen if they had a reading below ninety.

So when a patient has low oxygen, it can really cause a stress on vital organs because those organs aren't getting the oxygen they need to function normally. And the most serious harm that can occur is that your heart will have a heart attack or even stop if the heart has low oxygen. But besides the heart, other organs, when they don't get enough oxygen, can be damaged, like the kidneys or even the brain.

TS: 10:26

So there are a number of factors that can affect the accuracy of pulse oximeters. So doing your training, what did you learn about, you know, the things that might give you a wrong reading?

MS: 10:38

Yeah, so I would say that our training as a medical provider on a pulse oximeter was fairly limited. We didn't have a comprehensive training on it. think in general, we learned that what you need for a pulse oximeter to have an accurate reading is a good perfusion of the blood to the skin. So when your skin, when your fingertip is, like, very cold, it can have a less accurate reading. Or if a patient has maybe say low blood pressure, the reading can be lower. Another thing that I think maybe people were taught was if someone's wearing fingernail polish on their finger, it could lead to a lower reading. So I think those are probably the major things that we were taught.

TS: 11:27

So if we could go back to twenty twenty and the study that you led on racial bias in pulse oximetry measurement. So we want to talk about what motivated that study, which you alluded to. So back in 2020, it was the first wave of the pandemic. You were in the ICU. So what was different that made you start questioning this technology?

MS: 11:51

Yeah, so when we were in the ICU in twenty twenty, and we were caring for these very sick patients with COVID, we were basically seeing both the pulse oximeter readings and the arterial blood gas readings at the same time pretty frequently. And one thing that we started to appreciate was that we would see the pulse oximeter reading in a normal range, like I talked about, let's say, you know, ninety eight percent or something, which looks good. And then we would look at the arterial blood gas and it would be much lower, eighty eight percent or something that was very surprising. And we didn't really initially know what to make of that. Sometimes it just seemed like the reading on the pulse oximeter was normal and then the reading on the arterial blood gas was low.

MS: 12:45

And I think the serendipity was that because we had this group of patients with severe COVID and we're seeing the pulse oximeter readings and the arterial oxygen readings at the same time, we started to see this pattern, which I hadn't really appreciated before. And then at the same time, when we started to look at the, some historical literature on pulse oximeters and their inaccuracies, it was pointed out that in the past there had been a description that studies showed that pulse oximeters could be less accurate in people who were black or had darker pigmented skin.

And so when we heard about these studies from the past and then sort of saw from our own experience what we were seeing at the bedside, we put two and two together that like, I think that's what we're still seeing. And it was sort of a surprise to even think about this potential problem because these studies were older. They were studies from the nineteen nineties or early two thousands. Here we were in twenty twenty and hadn't really appreciated these older studies or the fact that these pulse oximeters could be inaccurate. But that's really what motivated our original study.

TS: 14:12

And to go back to the ICU, can you talk about a specific moment maybe that made you realize that something was wrong and what did you see that didn't add up?

MS: 14:23

We would see this patient in the ICU critically yell, look at their pulse oximeter reading on the monitor, seeing that it was in a safe range. And then we would receive the arterial blood gas report and see this discrepancy. And I remember I saw it and then a couple of my other colleagues saw one as well. And we were just sort of like scratching our heads puzzling. And once a couple of us sort of all said, hey, why are these readings inaccurate? That's really what got our heads scratching to try to get to the bottom of the problem.

TS: 14:57

So what's actually happening at a technical level that causes these inaccurate readings in patients with darker skin?

MS: 15:05

Our best understanding of what's happening is that as the pulse oximeter is shining a wavelength of light through the skin, the melanin, the pigmentation in the skin of darkly pigmented patients, ultimately changes how the light is sent through the skin. And in a way that I don't think the pulse oximeters were truly calibrated to account for. And so a pulse oximeter that is measuring how much light is sent through the skin is thinking that, it's OK. Like, the amount of light that's shining through the skin is OK, when in fact, it's not OK. Because what we found was the pulse oximeter was overestimating the oxygen levels. So what that meant was that, in a person with darkly pigmented skin, their reading on the pulse oximeter looks okay, but in reality, their oxygen level is low. So something was happening where, as the pulse oximeter was shining wavelength through the skin, it was miscalibrated in some way, sort of leading it to think that things are okay when actually they weren't.

TS: 16:25

And also just to clarify, you were mainly looking at black patients, but you know, for patients who aren't black, but have a darker skin tone, how are they impacted?

MS: 16:36

Yeah, so in the study that we did, we just looked at people who describe themselves as black and people who describe themselves as white. But one can extrapolate from our study in that people with intermediate levels of skin pigment would have sort of that effect would also be there. It might not be quite as significant, the inaccuracies, but we would expect someone with an intermediate level of skin pigment to have more inaccuracy in the pulse oximeter than someone with lighter skin pigment.

TS: 17:17

So in terms of the implications for patients, mean, you're in the ICU and you're looking at a pulse oximeter reading on a black patient. What goes through your mind now that you know what you know?

MS: 17:31

I just scrutinized that value quite a bit more and I approach my care of that patient slightly differently. I now have a higher index of, we use a term like index of suspicion, like at what point should we start to be worried about a patient? Like fundamentally, one of the things that we do as clinicians in the hospital is identify patients who are sick from patients who are well.

Who are the sickest patients who need the most care? And we try to make that decision based on all the data that we have access to. And one of those data points was the pulse oximeter value. If their saturation was normal, that was very reassuring that they were, maybe they were doing okay. And I think we, in the past, had put a lot of weight on that reading. If that reading is normal, that's very reassuring that the patient was okay.

MS: 18:25

But this study and then several others that have come out since this study has sort of raised concerns about the accuracy of these devices. It turns out that, you know, they're good, they're accurate, but they're not perfect. And so because of that, I've just taken the reading but just not let that value sway me, you know, particularly if other things that I know about this patient don't really fit, like if the patient is acting thick, breathing fast, if there are other signs that suggests the patient is sick, but the pulse oximeter is normal, I'm not putting maybe as much weight in that result as I would have in the past.

TS: 19:07

So I want to dig into the methodology for the study a little bit. So you had this hunch from the ICU. How did you go about actually proving it?

MS: 19:18

In the ICU setting, we often are drawing arterial blood gases because we use that information to titrate how much respiratory support we're giving to patients. And so because of that, we happened to have these arterial blood gases, which we don't always have. And so in that patient population, because we had the pulse oximeter value, followed by an arterial blood gas, we were able to compare those two results in a way that we often can't do because we don't have the arterial blood gas.

MS: 19:57

So we basically used the data that was being collected as part of patient care. Like we looked at the pulse-accented results that were recorded into the hospital chart, and we looked at the arterial blood gases when they were available and we found opportunities where the pulse oximeter result was recorded, followed by an arterial blood gas to see how those two results compared. And then what we looked at more specifically is we looked at instances where the pulse oximeter was reading in a normal range, but it turned out the arterial blood gas was actually low. So the true oxygen level was actually low. And we looked at how often that occurred in patients who described themselves as being white and then patients who described themselves as being black. So that was sort of how we set up the analysis.

TS: 20:58

How big was the sample size?

MS: 21:01

In the patients we analyzed at University of Michigan, we had just under thirteen hundred white patients and just over two hundred and fifty black patients. So that was actually a pretty big sample size. And that's, in general, a lot larger than some of the other pulse oximeter studies that we had in the past because we had this large data set of patient samples that we could analyze, particularly from COVID nineteen, we were able to get this large data set and do this analysis in a way that people hadn't really done before.

TS: 21:38

And then, so that was the initial study with the patients at the University of Michigan, but I understand you also did additional studies. So you're looking at veteran affairs data, other hospital systems. Can you tell us a bit more about what that revealed?

MS: 21:54

Yeah, because of the way we did this study and because of the way that healthcare data is now getting collected, there really was an opportunity to basically repeat the study in different patient populations. So members of our team, including myself, repeated the study using patients from the Veterans Affairs hospitals across the United States using the same general methodology, looking for opportunities where the pulse oximeter reading was recorded, followed by an arterial blood gas analysis. So we did that study in the Veterans Affairs Hospital. And then I think several other groups were able to do similar studies in their own hospital systems using their own data.

MS: 22:39

And one of the other things that came out as a follow-up study was looking at whether these differences in accuracy might have had differences in how care was provided. I think on the one hand, it's important to show, hey, this pulse oximeter is less accurate, but it's also important to show that this pulse oximeter inaccuracy is actually impacting how care was provided. And so some of the follow-up studies basically took it a step further, showing that this differences in pulse oximeter could be leading to differences in how care was being provided, whether it be oxygen levels provided to the patient or even potentially treatment for specific conditions that you might not actually treat a patient for unless you had a lower oxygen reading.

TS: 23:42

So that helped to confirm the initial hypothesis, the additional studies that you did?

MS: 23:48

Yeah, that's right. The studies that we did plus other people did really sort of show the same general results, which added sort of more evidence that this wasn't just a sort of a spurious finding just at our own center, but it was being widely seen across health systems.

TS: 24:08

And so when you step back and you looked at all the evidence, what did it tell you about how these devices were designed in the first place?

MS: 24:19

What I can imagine is that when these pulse oximeters were first being developed, they had to be, we use the word calibrated, which means we basically, we look to see what the pulse oximeter readings are and we train it to learn what that actually means in terms of what is the actual oxygen level. And if a lot of the original studies, when these pulse oximeters were developed, if they only included patients that were primarily white or not a very diverse population, one could start to appreciate why that device may not have performed as accurately in certain people. So I think a question mark is whether or not, you know, if these devices were originally developed in a more diverse patient population, would we have seen the same findings?

TS: 25:16

I read an interview that you gave where you sort of described the bias as sort of being baked into the design of the system, which I thought was quite interesting. And when people hear that, especially when people think of medical devices as objective, what is it that you want them to understand?

MS: 25:36

Pulse oximeters specifically basically gives us raw information about how much wavelengths of light are going through the fingertip. That's the objective information. But then what you ultimately need next is, well, what does that information mean? And in order to understand what that information means, we need to look at, in a broad group of patients, OK, this amount of light shining through the finger means this oxygen. And so that's how sort of this bias was sort of baked in, in that the way that the device was interpreting this objective information that it was gathering seemed to be where the biases occurred.

TS: 26:22

And what about skeptics who might say, there must be another explanation. So biological differences, for example, underlying health conditions.

MS: 26:32

Yeah, I think that some of that is a fair question. So one thing that we started out in talking about was in general, what do we know about pulse oximeters and what do we know about their accuracy overall? One thing I mentioned was that, you know, in patients who have lower perfusion of their blood to their fingertip, their reading could be less accurate. And so, you know, someone who says, well, well, the pulse oximeters were maybe less accurate in patients who are black and those patients also happened to have more ill health, they're more significantly ill. So maybe the pulse oximeter was just reading less accurately because of that. And I do think there is probably a bit of truth to that. But I think the evidence shows that it's really a combination of factors, including skin pigment and also sort of perfusion differences that are really magnifying the problem.

MS: 27:34

Something else that sort of came out of this, all this research is that pulse oximeters when they're being developed are being developed and tested in normal healthy individuals, not individuals who are in the hospital. So it may be the case that in normal healthy individuals, whether you be white or black, the difference in the accuracy or the difference in the performance is maybe small. But maybe actually in the hospital, when both of those groups are sick, you really start to see the difference. Because that sort of, there's that little difference is actually magnified now that, you know, this group of patients is sick. So in reality, what we really need are pulse oximeters that we know work really well in these critical situations.

TS: 28:28

And so if we can talk about the regulation, so these devices are regulated in the US by the Food and Drug Administration or the FDA. How do these devices get approved?

MS: 28:42

For a company to market a device and sell the device, the device has to undergo a approval process by the FDA. And as part of that process, the company has to provide data about the performance of the pulse oximeter when it's tested. And it's tested against the measurements are being taken at the same time that the arterial blood gas is being drawn to sort of check to make sure we have high accuracy.

So in the past, the FDA's sort of requirement was honestly fairly small. You know, the number of patients that the device needed to be tested on for evidence that the device worked effectively was quite low. And also the diversity of the sample of patients of people the device was tested on was also pretty minimal. And so for years, even though there was this regulatory body that was overseeing pulse oximeters and evaluating pulse oximeters before they were taken to market, frankly, the sort of the level of evidence that was needed to sort of demonstrate these devices were effective, I would say was fairly low, simply because the number of patients they needed to be tested on was low.

TS: 30:12

And you were saying earlier that there have been studies on pulse oximetry and the discrepancies going back as far as the nineties. Why do you think your study broke through when the early ones didn’t? And I know you mentioned the sample size. Are there any other factors that you think played a key part?

MS: 30:32

Yeah, I think in the past, when you reread some of the studies, these studies showed differences in accuracy of the pulse oximeter. But I think in general, around the time, some of the studies conclusions was, well, there is, you know, the small difference in accuracy that we're detecting, but maybe it's not clinically important enough for there to be a cause for concern.

And so I think one of the things that we did with our study was we really set it up to answer this important clinical question. When the pulse oximeter is reading in the normal range, how often is it wrong in a way that could potentially be harmful for patients? So I think our study really sort of set out to answer that really important clinical question. I think the sample size was larger. And then I think the timing was sort of you know, right, because people were really thinking about pulse oximeters. And then I think around the same time, there was a lot of additional awareness about disparities in populations.

TS: 31:44

And when the study came out in twenty twenty, it was published in the New England Journal of Medicine. How would you describe the response?

MS: 31:54

I think the response was overwhelmingly positive and people were grateful that we had published the study shed light on this problem that people hadn't paid attention to in quite some time. And I think, again, because people sort of recognize the importance of these devices, immediately wanted things to change basically, like how can we make sure that pulse oximeters are accurate and the pulse oximeters that are being used to make these critical patient care decisions, we can feel confident in those devices?

I would say that the response was very positive and then there was a lot of follow-up. I think if I recall, only a month or two after our study came out, there were some US senators that wrote to the FDA raising concern about this issue, asking for a really comprehensive evaluation of the processes that they were currently using to evaluate pulse oximeters. And there's been a lot of progress since then. One of the things that we've seen is that the FDA has proposed a revision to how pulse oximeters are gonna be evaluated before companies are allowed to market them.

TS: 33:16

Right, so they actually acknowledge that these disparities exist and they are clinically significant. Is that a fair summary?

MS: 33:25

I think that's right. I think at the FDA, based on meetings that I've listened to recently, there is a general consensus that these disparities exist and we need to do something about them.

TS: 33:37

And what about the manufacturers, the companies making these devices, how have they responded?

MS: 33:43

Well, initially they were very defensive about the study, you know, because we were sort of suggesting that these devices that they were producing were less accurate in a minority patient population. You know, it wasn't very good look to sort of be a company who was hearing that. And so they stood by their products and for a while sort of wanted to poke holes in our study, look for ways in which the study was wrong. And so I think it actually helped quite a bit when a number of follow-up studies started to show the whole thing. It sort of took a bit of the heat off of us.

But I think over time, what I have seen is support for efforts to make sure these devices are accurate. And, in general, from what I've read and seen, the companies have been supportive of the new FDA guidance, for example, for what sort of testing procedures should go in to the device before it's taken to market.

TS: 34:44

So you talked earlier about how this has changed your own practice. From a patient's point of view, say someone listening is you're sitting in a doctor's office tomorrow and the nurse puts a pulse oximeter on their finger. What do you want them to know?

MS: 35:01

I want them to know that a pulse oximeter is a very valuable tool. I think we're much better off having pulse oximeters than not having pulse oximeters. I cannot imagine what it was like to practice medicine before we had pulse oximeters, but like everything we have in medicine, you just can't rely on that piece of information without looking at the whole picture.

And so if the pulse oximeter seems like it's working and reading normally, but everything else feels off. I think it's important to not put the value of the pulse oximeter and, you know, focus on that value without considering the whole picture. And so I think if your pulse oximeter is reading just in that range of, you know, ninety two to ninety five, but you're feeling off, don't trust that pulse oximeter if other things don't seem right. I think that was, that was a lesson we learned in COVID is that, like, that number by itself does not give you the whole picture.

TS: 36:02

Are there any final thoughts you'd like to leave listeners with?

MS: 36:06

Yeah, I would just say that I'm grateful to the medical community for taking on this challenge. I helped lead the study in twenty twenty, but so many other people have also become involved. I've met people along the way who have repeated some of the analysis that we performed. I've met people who are working on developing new technologies to improve pulse oximeters. And so I'm super grateful that there's all these people across the world that are working on this problem because they recognize the importance. And so I just feel confident that we are going to solve it now that people are aware of it and recognize the importance of it.

TS: 36:50

Well, Michael Sjoding, thank you so much for joining me. It's been a pleasure having you on the show.

MS: 36:54

Thank you very much for the invitation.

TS: 36:58

That was Dr. Michael Sjoding, Associate Professor of Medicine at the University of Michigan. If you'd like to read the study we talked about, I've linked to it in the show notes.

We'll continue the conversation on pulse oximeters later on in the season when we speak to Professor Valenciaa Koomson from Tufts University. So be sure to look out for that.

Thank you for listening to this episode of Made For Us. If you liked it, let us know by leaving a rating or review on Apple Podcasts or Spotify. And don't forget to text such a friend or colleague who hasn't discovered the show yet. I'm Tosin Sulaiman, see you next time.