Google Launches Cloud Dataproc, A Managed Spark And Hadoop Big Data Service

Google is adding another product in its range of big data services on the Google Cloud Platform today. The new Google Cloud Dataproc service, which is now in beta, sits between managing the Spark data processing engine or Hadoop framework directly on virtual machines and a fully managed service like Cloud Dataflow, which lets you orchestrate your data pipelines on Google’s platform.

Greg DeMichillie, director of product management for Google Cloud Platform, told me Dataproc users will be able to spin up a Hadoop cluster in under 90 seconds — significantly faster than other services — and Google will only charge 1 cent per virtual CPU/hour in the cluster. That’s on top of the usual cost of running virtual machines and data storage, but as DeMichillie noted, you can add Google’s cheaper preemptible instances to your cluster to save a bit on compute costs. Billing is per-minute, with a 10-minute minimum.

Because Dataproc can spin up clusters this fast, users will be able to set up ad-hoc clusters when needed and because it is managed, Google will handle the administration for them.

“In this space, there is no one-size that fits for all,” DeMichillie said. “We think this is going to be a really important addition to our overall portfolio.”

KciNhA1_PKSnFIsuVnhhvwmCPGiLfaOfGhPfBAT0I7eykLJI3f1SQcyhRyZs1b5LOUe1yA=s2048

Because the service uses the standard Spark and Hadoop distributions (with a few tweaks), it’s compatible with virtually all existing Hadoop-based products, and users should be able to easily port their existing workloads over to Google’s new service.

DeMichillie and Google product manager for big data products James Malone told me Google is able to ensure the service’s speed thanks to its network infrastructure, but also because it patched a few Spark issues (related to the open source YARN resource manager the company is using for this product) and by building optimized images.

DeMichillie acknowledges that some people simply want to have full control over their data pipeline and processing architecture and are hence more likely to want to run and manage their own virtual machines. In his view, Dataproc users won’t have to make any real tradeoffs when compared to setting up their own infrastructure.

Unsurprisingly, Dataproc is also integrated with the rest of Google’s cloud services, including BigQuery, Cloud Storage, Cloud Bigtable, Cloud Logging and Cloud Monitoring.