Apple Watch’s secret and changeable data algorithms are an issue for researchers due to inconsistent data.
A Harvard biostatistician is rethinking plans to use Apple Watches as part of a research study after finding inconsistencies in the heart rate variability data collected by the devices. Because Apple tweaks the watch’s algorithms as needed, the data from the same time period can change without warning.
“These algorithms are what we would call black boxes — they’re not transparent. So it’s impossible to know what’s in them,” JP Onnela, associate professor of biostatistics at the Harvard T.H. Chan School of Public Health and developer of the open-source data platform Beiwe, told The Verge.
For the most part, his teams use research-grade devices that are designed to collect data for scientific studies. As part of a collaboration with the department of neurosurgery at Brigham and Women’s Hospital, though, he was interested in the commercially available products… So, they checked in on heart rate data his collaborator Hassan Dawood, a research fellow at Brigham and Women’s Hospital, exported from his Apple Watch…
Because the two exported datasets included data from the same time period, the data from both sets should theoretically be identical. Onnela says he was expecting some differences. The “black box” of wearable algorithms is a consistent challenge for researchers. Rather than showing the raw data collected by a device, the products usually only let researchers export information after it has been analyzed and filtered through an algorithm of some kind.
MacDailyNews Take: If Apple wants to keep the Apple Watch algorithms secret for competitive reasons, the company should provide a way for researchers to access the devices’ raw data. Problem solved.
My 🍏Watch is my personal medical monitor. I don’t really CARE that the information can’t be reliably shared. What I DO care about is that it is reliably reported. What is my Blood Oxygen level, what is my ECG, what is my pulse, how many steps have I taken … important stuff (to me). Soon (I hope) Apple will introduce a blood-sugar monitoring feature … and I get to update my current watch. BP would be nice, but sugars are more important. To me.
But the problem they are reporting hits directly at your point – it will change depending on which watch you’re using, and which version of WatchOS you’re running. Until this is changed, it will not be accepted by the medical community. And researchers are stuck using horrible and expensive research grade ‘gold standards’ like the Philips Actiwatch.
Or how about normalize your data to a single release version. Not doing so shows what terrible hacks these ‘scientists’ are. Even if the algorithm doesnt change, the rest of the software could change how things work. You want to normalize to the same model, the same version. That is just basic research 101. This smells of planted bs.
This is a horrible way of doing things – ideally for most things you might want data on (fitness / activity levels, heart rate data, etc.) you want to look at how they change longitudinally (i.e., in disease or following intervention or in recovery). Without knowing when and how the underlying algorithm has changed you can’t do that. And if you’re relying on people sharing their data from their own devices that you can’t control, you’d need to ask them watch model, OS, etc. and figure out when that’s changed, etc. It’s not surprising that Apple (and FitBit, and others) keep their algorithms secret, and most don’t release raw data either. And hence researchers are stuck using expensive wearables that are less functional than consumer products. Bummer.
The only thing horrible is your lack of basic science norms and reading comprehension. If you keep the same model an operating system version, you have a normalized apples to apples data set. If you change any of those variables, you do not. It’s good practice to normalize equipment.
I think that’s his point. Since the researcher is not providing the device and relying on personally purchased devices, normalizing equipment is very difficult. It may be unreasonable to expect the users to agree to not updating their OSes for the duration of the needed data set.