This study has good and bad news about fitness trackers

Alan Martin May 25, 2017

We know that not all fitness trackers are created equal. A study last year, for example, found that the Apple Watch was the most effective wearable at tracking heart rate, but it appears there’s something they all have in common: they’re pretty lousy at tracking calories burned.

This study has good and bad news about fitness trackers

A new study from Stanford University sought to assess the accuracy of seven fitness trackers on two separate metrics: heart rate accuracy and energy expenditure. The good news is that in terms of heart rate measuring, all seven trackers – the Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn and Samsung Gear S2 – were pretty reasonable: six of the devices were off by around 5% most of the time, which is accurate enough for non-medical use.

Now the bad news: energy expenditure was all over the place. While being monitored via indirect calorimetry (a system that calculates metabolism by measuring oxygen and carbon dioxide on participants’ breath), the researchers found that amongst the 60 volunteers, not one of the wearables was within 20% accuracy. On average, the devices were off 27% of the time, with the least accurate device proving to be the PulseOn, which was off 93% of the time. fitness_tracker_energy_expenditure_wrong

In order of performance, the Apple Watch and Fitbit had median errors of a little over 20%, followed by the Microsoft Band and Basis Peak, with the PulseOn lagging behind in last place. The Mio Alpha 2 and Samsung Gear S2 managed to dodge the test, on account of not providing data to compare every minute.

A little extra good news: the way they were wrong tended to flatter the gym bunnies. The study’s co-author Anna Shcherbina told USA Today that the devices had a tendency to underestimate calorie burn during light exercise, and to overestimate it during more strenuous workouts.

“You can’t really trust them at this point,” she added. “They can be a useful guideline, but still, you can’t rely on the device 100%.”

There was another interesting nugget of information in the study: the devices tended to be less accurate for men, those with higher body mass indexes (BMI) and people with darker skin than for Caucasian women with a healthy BMI. “So, for those for whom it might matter the most, who are trying to lose weight, the error was actually greater,” Euan Ashley, associate professor of medicine at the University of Stanford Medical Centre told NPR. He speculated that this might be down to a narrow group of people tested when the devices are undergoing R&D – though, it’s worth remembering that this was a very small study of just 60 people.

Both Fitbit and PulseOn have defended their devices’ accuracy in the face of the research. “Overall, the success of Fitbit products comes from enabling people to see their overall health and fitness trends over time,” Fitbit said in a statement. “It’s these trends that matter most in achieving goals.”

PulseOn CEO Jaakko Hattula questioned the methodology of the study, writing in an email to USA Today that “we think the excess error reported in (energy expenditure) is not representative in this study, due to this methodological error.”

You can read the full study in the Journal of Personalised Medicine.