If Inaccuracy Were Illegal, The Feds Would Have To Regulate Most Health Gadgets

I wore a calorie counter armband from BodyMedia that told me I was burning over 3,000 calories a day. If that were true, I would be emaciated enough to play Gollum in the next Lord Of The Rings sequel.

The accuracy of consumer health gadgets varies widely across the spectrum, yet this hasn’t stopped the FDA from suspending popular genetic testing company 23andMe over concerns about the quality of their diagnostics. 23andMe may be at fault, but so is a large part of the entire health tech industry.

The FDA is scared that 23andMe could cause a stampede of unnecessary and invasive medical procedures from consumers who get false readings on life-threatening abnormalities. Specifically, women who test positive for a gene associated with onset breast cancer, BRCA1, have been known to demand mastectomies against doctors’ advice, even though “false positives” are quite common.

23andMe advises their consumers to follow up breast cancer concerns with more accurate diagnostics, but the FDA is rightly worried it could trigger neuroticism that won’t be quelled by more sensitive tests.

We can debate whether 23andMe is at fault, but if they are, so is the massively expanding industry of wrist pedometers, scales, blood tests and sleep trackers. Trust me, dieting information causes irrationality every bit as much as genetic abnormalities.

Wild Variation

At least for my body, some health trackers are inaccurate. Every day, the Basis watch, Nike FuelBand, Jawbone UP, BodyMedia, and Fitbit gave me daily readings for calorie output that could differ by over 1,000 calories (others have found similar variance).

To calibrate them to a more accurate measure of calorie burn, I got a metabolic rate test at Breakaway Performance in San Francisco. The test involves running on a treadmill while a mask measures energy output from carbon dioxide expelled from my lungs. Here’s how three popular trackers (from last summer) compared to the 140 calories that Breakaway said I burned:

Jawbone UP: -13% (123 calories)
Nike FuelBand: -8% (129 calories)
Basis: -3% (136)

Now, a dozen calories may not sound like a lot, but over the course of a day, it can mean the difference between gaining and losing weight. On the day I tested, here was my total calorie burn:

Jawbone UP: 2,072
Basis: 2,440
BodyMedia: 2,753
Fitbit: 2,408
*Nike only gives exercise-induced calorie burn.

I’ve had similar luck comparing fancy body-fat, percentage-measuring scales  to a hydrostatic “dunk test”:

Fitness Wave hydrostatic (18.9 percent) vs. Withings Smart Body (22.8 percent).

For men, a 4 percent delta can mean the difference between “average” and “fit” — not to mention much bigger love handles.

I should note that this is not a test of what device is most accurate. I don’t know and it certainly depends on the individual user. For instance, for those who ride a bike, most health trackers, such as the UP, FuelBand and Fitbit, can’t tell if you’re struggling up a San Francisco behemoth or gliding on a professional track. I also own a walking desk, and health trackers have difficult recording steps when I’m typing (instead of swinging my arms by my side).

I had originally planned a much more thorough review of the devices’ respective accuracy of steps, calories and sleep, but had trouble getting data from all the gadgets except BodyMedia and Basis (the only two that give minute-by-minute to consumers). I’m going to re-do my test on the next generation of devices soon.

Suffice to say, while I don’t know what is most accurate, I know most of them have to be misleading.

Regulate All Or Regulate None

Most consumer device companies begrudgingly acknowledge they aren’t very accurate. When I confronted Jawbone with my preliminary findings, a spokesman wrote to me, “UP is focused on overall lifestyle and helping people understand their baseline of their core activity throughout the day.” In other words, if they had done more scientific comparisons, they weren’t telling me.

23andMe, too, warns users in bright highlighted letters to get “clinical tests” for BRCA if they have a strong family history of breast cancer.

Screen Shot 2013-11-25 at 1.31.13 PM

The same goes for the rise in blood test companies, though our ability to test for the presence of nutrients, such as Vitamin D, is still developing.

Perhaps 23andMe raised the ire of the FDA because of its aggressive defense of releasing BRCA information. They write blog posts citing research on how false positives really don’t cause regret in women. A bold move.

Regardless, like 23andMe, the staple of every health tech company’s marketing department implies changes in lifestyle. The FDA has chosen to grant most of these gadgets and apps some leniency with the less stringent Class II regulation. But, it’s hard to see a difference between genetics and exercise, especially if a calorie counter leads users to dramatically cut their food intake.

There are a few ways in which consumer response to health trackers go could go awry.

First, as The Switch points out, many of the implied recommendations from 23andMe still have to go through a doctor (like prescriptions or advanced screenings). In contrast, many people adopt diets and exercise habits without having to consult a physician. They may buy a standing desk and run in the evenings, without knowing that it’s crucial to move throughout the day.

A sedentary lifestyle kills many of thousands of people each year, so getting the exercise part of one’s health right cannot be underestimated. It’s not hard to meet the steps or calories goals of some health trackers and still be at risk for diseases associated with a desk job.

Second, there have been calls to regulate health trackers and apps for some time. BodyMedia, the band that gave me the most eye-opening results, boasts that is a FDA-approved device and is used in research projects. Yet, executives at BodyMedia have admitted to me that all devices have difficulty accurately measuring activity (cross-training exercising like Crossfit throw them for a loop).

It should be noted that BodyMedia goes pretty far at giving actionable advice. It offers a detailed calculator about how to lose weight by combining exercise and diet, yet the calculator ignores that not all diets are the same.

Inaccuracy could seriously mislead patients who need a calorie deficit, or discourage those who are led to believe they’re not exercising enough.

Personally, I’m not opposed to the FDA forcing tech companies to perform due diligence on their devices. Maybe informational products just shouldn’t be regulated. But let’s not pretend that genetic information makes consumers any less irrational or reckless than dieting.