Searching For Truths In Big, Enormous, Massive Data

We have so much data, mountains and mountains of data. In fact, there’s so much, the name ‘Big Data’ doesn’t even do it justice. Maybe we should call it humongous data or at least enormous data. Yet for all this data, it seems we are lost in those piles of data, no closer to the answers that are supposed to solve our most fundamental business and social problems.

Having this enormous amount of data is after all supposed to give us a huge competitive advantage. In fact, MIT professor and author Andrew McAfee is fond of saying we would be better off listening to the data instead of the organizational HIPPO, the highest paid person’s opinion. In most organizations, McAfee says, the person at the top who has risen there because of their depth of experience, uses that experience and intuition to make the company’s biggest decisions, but he says we don’t have to look to the HIPPO necessarily because the answer can be found in the data.

I’m not convinced it’s quite that simple though as I’ve noticed several instances of data usage recently where it wasn’t an easy matter to just look at the data. Data is dense and complex and sometimes when you look at it, it takes some cajoling to get the answer or it takes knowing what questions to ask –and if you are looking for answers to hard questions, there is not always a clear path.

When I was at Web Summit in Dublin recently I interviewed a panel of data scientists and I asked them if there were danger in bringing data discovery to the masses because without the expertise the data scientist brings, they could be asking the wrong questions or coming to false conclusions. As you would expect the data scientists came down on the side of data science.

When I asked the same question to a panel of big data vendors, while they acknowledged that danger existed, they came down on the side of the non-technical business user.

Nothing too surprising there, but can we rely on data to make better decisions or do we just enter a twilight zone of data piles with all of this data and so few answers, where we are forced to beg at the desks of the data scientists to get the answers we need? Isn’t this all supposed to make us faster and more agile, not put another layer between lines of business and the information they need? Data can give us answers to specific questions and even surface answers we didn’t know existed, but there is no magic way to get there yet.

Sure, there are tools that are beginning to simplify things a bit and helping us visualize all that data, but I still can’t help but feel we are like the cops looking for a murderer, knowing that the data is trying to point us to the killer, but not knowing how to get there. We have all these clues, but the data doesn’t somehow take something complex and difficult and make it simple and easy.

In business, there is a similar level of frustration. We have figured out how to collect the data and we can process it and clean it, and make pretty pictures related to it, but it doesn’t feel to me at least that we are any closer to getting the answer than the cops.

One thing I do know for sure, there are answers in that data if we only ask the right question or it deems to show itself to us somehow, but getting there remains a monumental challenge which many companies are working to solve. Yet, I can’t help but feel that everyone is dancing around the problem, looking at it from a number of different angles, but never quite getting to the point of solving it.

Software has potential to change the way we do everything, but when you are dealing with massive amounts of data, and you are trying to figure out answers to complex questions, there are no easy paths to get there, at least not yet. Maybe if we study the data.