These days, you don’t have to look hard to find big data. Even a little startup can produce gigabytes a day, and a company the size of Instagram can easily generate 500 terabytes a day. If you’re like many companies I talk to, you’re sitting atop an ever-growing mountain of data, scratching your head and wondering, “Okay, I’ve got big data. Now what do I do?”
You don’t win gold medals just for having data. The real winners are those like Amazon and Netflix that find ways to leverage their data better than the competition. Without a game plan to turn your data into revenue, you may as well scrap your badass Hadoop cluster and the petabytes of data it contains.
On the other hand, if you find ways to use that data better than your competition, you just might join the ranks of the big (data) boys like Amazon and Netflix.
So, how can you start to turn your data into cash? For most companies, there are two ways they can wield their data assets to create unfair competitive advantages: data-driven processes and data-driven products.
In the era of big data, business analysts entering equations into Excel and running ad hoc queries in a SQL database doesn’t cut it. The new era demands a new breed of intrepid data explorer, one competent at using tools in both the small and big data worlds.
Dubbed a data scientist, this next-generation data geek knows enough about traditional BI tools, query languages, statistics, and machine learning to be considered armed and dangerous.
Good data scientists can help you do everything from figuring out what is and is not working in your products (as they are used at Zynga), to creating predictive models that let you peer into the future so you can make better decisions today (as they are used at @WalmartLabs).
Here are a few concrete examples of ways data scientists can help you:
The flip side to using data to drive your business processes is using data to enhance the functionality of the products you make (which doesn’t apply to all products, for example, toothbrushes and pillows!). Some companies accomplish this by packaging data into a useful, insightful product they can then sell to other companies.
Twitter, while not a data product itself, licenses its data to providers like DataSift, who then go on to create a data product that companies gobble up for the insight it provides them. Some media companies package up their audience viewership data into products they turn around and sell to channel programmers and content creators. And so on.
The majority of companies building data-driven products, however, don’t create and sell pure data products. Rather, they use data to make their existing products more efficient, more intelligent, or more insightful in some way that directly or indirectly generates additional revenue.
Here are some real-world examples of the ways data is being used to drive intelligent and insightful features inside existing products:
You Too Can Be Data-Driven
If my descriptions of data-driven processes and data-driven products has you salivating, but you’re still wondering how you can go from mountains of senseless data to piles of cold hard cash, then I’ve put together some concrete recommendations that should help get you started.
Capture everything centrally. In this day and age of plummeting storage costs and ubiquitous (free!) big data stores, if you’re not capturing every single bit of data, you’re doing something wrong. I often tell companies that while you can always ignore data you have, you can’t analyze something you don’t have. Unstructured and semi-structured data stores let you store data in its raw format now and pay the cost of extracting structure only when and if you need to. So there’s no excuse for not storing transactions, interactions, behavioral data, sensor data, user-generated content, log files, and whatever else you can get your hands on.
Get yourself a data scientist. If you’re a startup, you need at least one data scientist on your team, or someone who can double as a data scientist. If you’re a larger company, you need a whole team, and it may be easier to train from the inside rather than hire new data scientists. Data scientists can sometimes be trained from strong business analysts or those with an excellent background in BI and SQL. Data scientists need to be carefully equipped with proper tools and access to company-wide data so they can answer ad hoc questions, perform exploratory data mining, support BI teams, and help with data productization. A great data scientist will help you figure out the questions you need to ask in order to advance your agenda. He or she will also be looking for new ways to take advantage of all the data your company has access to.
Productize your data. Any company with proprietary data should strongly consider using that data to build new products or drive data-driven features inside existing products. Any company who has a desktop, mobile, web, server, or media-based application has proprietary data (which means, in this data and age, most companies have proprietary data!). Companies, especially in advertising and retail, have made millions or even billions of dollars of incremental revenue by using data to drive intelligent features in their applications.
If you’re a B2B SaaS vendor, providing your customers with self-service reporting is one simple way you can translate data into a product feature that indirectly drives additional revenue. If you’re an e-commerce platform, using all the data you have at your disposal for recommendations and personalizations can drive substantial incremental revenue. If you’re a consumer app, using data to make your application smarter can lead to higher usability and better engagement. Having someone on staff to think about what kinds of features and products you can create from your data assets is the first step toward data productization, but ultimately, you will need engineering resources that can turn that data into features and products.
The Data-Driven You
Big data isn’t about the data, per se, it’s about finding ways to use that data to drive business processes and product functionality within your company. The meteoric rise of data science in the past few years is a testament to the fact that data is the currency of the 21st century. If you don’t do anything with your data, you’re at a severe competitive disadvantage.
But by taking a few simple steps, such as capturing all the data you can get your hands on, making sure you have at least one data scientist, and working towards data productization, you can ensure that you’re effectively “spending” all that currency currently piling up in your data warehouse.
Precog is a data analysis platform that helps companies productize their data assets. Precog allows companies to store, integrate, and analyze large volumes of measured data. Featuring the industry-leading implementation of Quirrel, the “R for big data” statistical language, the Precog platform allows companies to create analytics or predictive models that can be deployed as new products or features inside existing products. Companies use Precog to build analytics and reporting into their applications, to turn existing data assets into...