Finding The Elusive Big Wisdom In Big Data

Next Story

‘The Interview’ Nets $15 Million In Online Sales During Opening Four Days

You probably already know ‘Big Data’ is top of mind heading into 2015. How could you not? You are hearing about it constantly from vendors and journalists alike (guilty as charged). And you know what that hype says, right? Big data is going to provide all the answers, make your companies run more efficiently and help you make brilliant, data-driven decisions that give your organization a sharp competitive edge.

To some extent that’s true, but like like any over-hyped technology, many companies find implementation is hard and the reality is very different from the hype. They may have figured out effective ways to collect and process the data, but putting it to work it to make better decisions is another matter. These companies are finding a key missing link between big data and big understanding, and if they don’t find a way to resolve this, they will be left with a big pile of confusing data with few insights.

As one Silicon Valley insider told me recently, while big data collection and processing has gotten a lot of attention lately in terms of startup activity and funding, there is still a huge gap between expectations and results. As this person put it, “Big data hasn’t yet translated into big knowledge, big insights and big wisdom.” In their estimation we still have a ways to go before that happens.


We want to believe getting value from big data is as simple as pouring in the data, running a program and getting insights, but in fact, it’s much more complicated than that. Pam Baker, who is is author of the book Data Divination: Big Data Strategies says while there are actually clear cut instances where data points to a direct answer, this isn’t always the case.

“Data can give us definitive answers in many, many cases. For example, predictive analytics can accurately forecast when an airplane or water system part will break and also dictate to us when exactly we should replace it to get maximum use out of old part but before old part actually breaks,” Baker explained.

But she added, “There are also many cases when you can’t get a definitive answer, but you can choose from several possible actions or even choose to take no action. Just depends on what you’re working on,” she said.

Baker is right that some data-driven decisions are much more subtle and require, as Bruce Springsteen once sang, just a little of that human touch. People can help by developing sound metrics and powerful algorithms. They also have to figure out how to make the best use of what the data is telling them. Sometimes it’s straightforward, but often it’s not.


Further, we would love to believe that big data will give business users direct and instant access to that data so they can make the best decisions magically on the fly. Unfortunately, the tools we have today don’t offer that level of sophistication just yet.

To help resolve this issue, we need more data experts to help us process data and find answers in the vast amounts of information. Keith Rabois, an investment partner with Kholsa Ventures who has invested in big data companies such as Parstream says companies need data scientists whose role is conducting sophisticated analysis, something your average line-of-business user is just not capable of doing.

Rabois says you want these data scientists creating applications and algorithms and doing the heavy data-science lifting, but in companies that actually have data scientists they don’t always have time to do it all, partly because they are spending time doing less sophisticated analysis that doesn’t take full advantage of their skill sets.

Rabois says in a best-case scenario, the data scientist has developed tools to help distribute  analysis across the organization among the different parties who require answers. What we don’t want to do in a day and age where we need answers quickly is to create a guru bottleneck where we go to an expert to get the answers, then wait for the results.

The problem is that even when smart people develop highly sophisticated algorithms, they don’t always give definitive answers to complex problems. It’s simply not possible to factor in every option or to take into account certain factors that are more difficult to measure.


If you want a good example of this, look at baseball where you can have two similar players on paper that produce very different results. Stat geeks will tell you that the algorithms they created over the years called Sabermetrics will provide all the information you need to identify a good baseball player to fill a particular role. They have created a whole range of them like Wins Above Replacement (WAR), a statistical measurement, which according to FanGraphs is defined as: “If this player got injured and their team had to replace them with a freely available minor leaguer or a [less talented] player from their bench, how much value would the team be losing?” They measure this difference in wins using a complex set of criteria.

There is little doubt that these sophisticated kinds of metrics can help compute the value of a players more accurately, but it can’t measure everything like how well he plays under pressure, how hard he practices, what kind of leader he is or how well he gets along with teammates. All of these things matter too and are much harder to quantify.

People who believe in pure statistical measurement will tell you everything can be measured, and that’s mostly true, but I’ve seen many cases of players that look similar on paper, yet don’t fill a particular role nearly as well as another player who was there before them in spite of their statistical similarities.

Applying this to business, human resources professionals could encounter an analogous situation with candidates for an open programmer position. You can have two similarly skilled professionals vying for the same job, but one may have people skills and work well with their co-workers and the other might be surly and difficult with poor communications skills, intangibles that aren’t going to show up on a resume. Even with a lot of data, it’s difficult to take into account every possible outcome, especially when it comes to people.


Any good doctor would tell you that even two patients with identical symptoms would likely require different treatments based on the individual variables of each person such as age, weight, other health issues and additional unique factors.

Consider the use of IBM Watson, the intelligent analytics platform, in medicine. When I explained to a friend recently that some doctors are using Watson to help with diagnostic and treatment decisions, he bristled. He didn’t want a machine deciding his medical treatment. It’s a valid concern, but in this case, Watson doesn’t give an answer like a trained seal that the physician blindly follows. It offers some options based on the evidence, what it knows about the patient, the symptoms and the current research on the matter (not unlike how a physician actually works).

As I pointed out, busy doctors can’t possibly do their job and keep up with every bit of research in their fields. There’s simply too much out there (and that’s a good thing). That’s where Watson comes in. It can filter through the most current research for the doctors much faster than people possibly could, but at some level it still requires the nuance and understanding of the physician to decide on the exact treatment direction. It’s what I like to call the art in the science. The knowledge takes you so far, but the final arbiter is the physician, not the machine.

Businesses are very likely going to face similarly unclear outcomes where people have to step in, use their training and make a choice with help from the data.


Machines can sometimes find answers and can give us insights it would have taken people years to figure out on our own. Baker points out, for instance, that big data has helped us find answers about diseases, such as cancer, that human researchers never even thought to look for. “We may never have found appropriate treatment (or at least not for many years) if big data hadn’t found that information for us. My point is that big data [can be] pretty darn accurate,” she told me.

What’s more, she believes machine learning will reach a level of sufficient sophistication in the not-too distant future where machines may be making more decisions for us simply because our brains can’t possibly keep up with all of the available information.

She is probably right, but for now, it seems, the ability to collect and process that data has gotten ahead of understanding what it all means. As Baker noted, predictive analytics are improving all the time, and sometimes the data points straight at an answer, but it still remains a complicated mix of human and machine and how this all comes together is very much a work in progress, even as the technology marches forward.

Until we find that balance or it tips sharply in the favor of machines, we face a big wisdom gap and that will take some time and technological advances to fill.

Featured Image: Darpa/Wikimedia Commons UNDER A Public Domain LICENSE