The Lab vs. The Real World: product testing is hard


Unless you’ve been living under a rock for the last couple months, you know that Toyota has had problems with sudden acceleration of some of its vehicles. Apparently, the root cause of the problem is still unknown, which is a little troubling to the average consumer. Toyota claims to be doing everything they can to investigate, but that doesn’t seem to be enough. Now everyone and their brother are suddenly product design engineers and have the gall to tell Toyota what to do and how to do it.

Witness this opinion piece in the LA Times by David M. Cummings entitled “Haven’t found that software glitch, Toyota? Keep trying“. Mr. Cummings worked on the Mars Pathfinder project for NASA, so he has some credibility to his name when it comes to software design and product testing. But his opinion piece seems to completely miss the point of the PR nightmare that Toyota is dealing with.

Mr. Cummings says “I’m still skeptical when I hear an engineer declare a complex software system to be bug-free based on laboratory testing.” I admit I haven’t been following the Toyota situation very closely, but I’d be floored if anyone at Toyota has gone on record to say that the millions of lines of embedded software used in Toyota vehicles is “bug free”. I’d be surprised if any professional software developer anywhere would have the gall to claim any code bug free. Code is complex. The systems on which the code runs are complex. The number of variables affecting the execution of that code are even more complex.

As Mr. Cummins observes at the beginning of his rant, “Toyota’s chief engineer testified to Congress that the company has done extensive testing on its cars’ electronics and believes they are not the cause of the sudden acceleration.” To claim that the software is not the cause of the sudden acceleration is not the same thing as claiming that the software is completely free of bugs!

What really sticks in my craw about Mr. Cummins complaint, though, is that he doesn’t appear to appreciate the audience that Toyota is addressing. Sure, some of the people following this situation are engineers, and they may well be interested in the details of the testing methodologies, or specifics of lab results; but most people are average consumers with no interest whatsoever in the science of this issue. They just want to know that the problem has been identified and fixed.

When I provide technical support to people, I don’t go into the specifics of memory registers, or heap and stack overflows, or any of the other things that happen to cause problems. I tell people that something went wrong in very simple terms. They trust me to know what I’m doing, else they wouldn’t have asked me to help them to begin with. They don’t want to get into the nitty gritty, else they’d pursue a job in technology on their own. I think Mr. Cummins needs to remember that not every Toyota owner, and certainly not every member of the U.S. Congress, is an engineer, or has an engineer’s mentality.

Mr. Cummins final remarks are very sound, though: “this should serve as a wake-up call to all industries that increasingly rely on software for safety.” I agree whole-heartedly. The world is only going to continue to get more complex. We need to build safety and reliability testing into every facet of product design, and not trust the tool chains we use to build software.