Pivotal Open Sources Key Parts of Its Big Data Suite

Pivotal announced today that it was moving three core pieces of its Big Data Suite to open source, while continuing to offer advanced features and support in a commercial version.

The three components moving to open source are GemFire, the platform’s in-Memory NoSQL Database, the enterprise SQL on Hadoop component, HAWQ and the suite’s massively parallel processing (MPP) analytics database, Greenplum DB.

One of the reasons it felt comfortable moving in this direction with the Big Data Suite was its success with Cloud Foundry, an open source Platform as a Service offering that followed an identical model of open source and commercial versions.

“We have had a lot of success with Cloud Foundry and we’ve been able to monetize it,” explained Sundeep Madra, VP and GM of the data product group at Pivotal. The company felt it could build on this success by taking the same tack with the Big Data Suite.

Pivotal was hearing from customers that they wanted the flexibility of open source software in other areas, which also helped pushed the company to make this move. “One of the things we’ve realized is that open source is critical to enterprise buying patterns,” Madra said. Pivotal has actually been seeing open source show up as an RFP requirement, and companies have expressed a need for freedom from vendor lock-in.

“There is definitely a notion that if you are a customer using an enterprise distribution of open source, you’re not locked into anything,” Madra explained.

What’s more, customers using open source software don’t have to wait for the company to get around to developing a required feature. They can work with the company or a third party or build it themselves. “Customers can take control of the project and solve it on their timeline,” he explained.

Pivotal Big data suite

It’s worth pointing out that Pivotal is a joint venture of General Electric, EMC and VMware, but it operates independent of these organization. The idea was to create a company that could make its own way and use technology in ways that the parent companies might not have the agility or creativity to do.

This type of move is precisely what they had in mind when they spun out Pivotal. Even though tools such as Greenplum DB were developed at great expense, Pivotal is showing it isn’t afraid to try new approaches such as moving these components to open source.

“There’s a reason Pivotal was spun out. We wanted to operate like an independent company and the stuff we are doing is only available because we are independent. This is our thinking that this is the best approach. We have our own board and our own desires and goals as a company,” Madra said.

He knows there is some risk inherent in a move like this, and that third parties can build products on top of the open source platform that compete directly with Pivotal. In fact, he hopes that’s exactly what happens, and he’s not afraid of competing offerings.

“We want to see an ecosystem build around those products. We aren’t going to put it in open source and walk away,” he said. He hopes to see a vibrant community develop around it, and while Pivotal will likely be the primary contributor to the project, it welcomes input from customers and other who have a stake in these tools. It also understands that some of these products could compete directly with Pivotal.

Pivotal also announced today it was joining the Open Data Platform, an alliance of 14 companies that includes Hortonworks, GE, IBM, Verizon and others to offer a single enterprise version of Hadoop.

The idea is that you have this common core set of agreed upon Hadoop components on top of which you can build solutions that are guaranteed to work with it. Pivotal and the members of the alliance have seen Hadoop become fragmented and this organization is an attempt to bring that under control.

“This is not a Pivotal initiative. It’s an industry initiative,” Michael Cucchi, senior director of the Pivotal data product group said. “This should happen because it makes it easier for the ecosystem to advance,” he explained.

Both of these moves are an attempt to create a more open platform on top of which others can develop. While Pivotal loses some control by moving in this direction, it offers customers a more flexible and open approach, and that could pay off more in the long run than trying to keep things under tight control.