BigQuery, Google’s SQL-based big data analytics service, is getting an update today that, among other things, will make it easier for users to avoid runaway query costs and to stream large amounts of data right into their databases.
Google is also introducing audit logs for BigQuery as a beta. This feature will make it easier for IT admins to track what their users are doing on the service.
With Google’s new BigQuery custom quotas organizations can now set up project- and user-level limits for how much data their queries can process every day. Those levels can be set in 10 terabyte increments for user-level quotas and 50 terabyte increments for project-level quotas. Google charges $5 per terabyte processed in BigQuery.
“The new granular cost controls in BigQuery will be a welcome addition for those accustomed to a traditional, fixed-size cluster, where cost is also fixed and scaling is complex,” Google technical program manager Tino Tereshko notes in today’s announcement. “Those accustomed to a traditional fixed-size cluster – where cost is fixed, performance degrades with increased load, and scaling is complex – may find granular cost controls helpful in budgeting your BigQuery usage.”
When it comes to figuring out cost, Google now also makes it easier for BigQuery users to see how their queries are executed on the service. With Query Explain, they can now get a graphical overview of your query that shows where you may have performance bottlenecks in your queries.
The other BigQuery feature Google is improving with this update is the Streaming API. The company is now making it possible for its users in the European Union to load datasets that currently reside in the EU into BigQuery. Streaming users now also won’t have to wait for the service to “warm up” before their data becomes available for analysis in BigQuery after streaming the first row into a table. This warm-up period could sometimes measure a few minutes, but now, the data is available immediately.
If you regularly use the Streaming API, Google now also offers a new feature called ‘template tables.’ With this, an existing table can function as a template for streaming insert requests.
Of these update, the ability to better manage query cost will likely have the most immediate impact. Google is trying hard to get more enterprise users to use its cloud services over those of its rivals Amazon and Microsoft — and enterprises like to have this kind of cost controls in place for their employees.