Confluent, the company that built a streaming service on top of the open source Apache Kafka project, has always been about helping companies capture streams of data. First it was on prem, then later in the cloud. By moving the streaming service to the cloud, it was able to abstract away a lot of the complexity related to managing the underlying infrastructure.
Today, at the Current conference, the company is introducing a new tool called Stream Designer to make it easier to build a streaming data pipeline in a visual workflow. Users can easily connect a set of data components to build a customized stream of data, and Confluent handles the coding in the background for them, essentially moving the abstraction up the stack from infrastructure to design.
Company co-founder and CEO Jay Kreps says that while it’s still aimed at developers, the goal is to put stream design within reach of more people than Apache Kafka experts.
“We’re still serving developers of all types, but this makes it something where you can just click and build a data pipeline that connects things, transforms data, sends things from place to place. And it inherits and works on the same underlying streaming infrastructure. That means all the data streams are reusable, they’re all real time and they’re all horizontally scalable,” Kreps told TechCrunch.
He says that although the program writes the underlying code for the developer, it is still accessible for developers who need to work with it. “One of the cool things about how we’ve done this, is even though you don’t have to write any code, all the code is there. You can see exactly what it’s doing under the hood. You can see exactly the kind of transformations in SQL that we’d be doing on this data, so it works with the existing developer tool chains if you need to do that,” he said.
Confluent Stream Designer example. Image Credits: Confluent
For Kreps, this is all part of the evolution of Kafka, a project originally conceived at LinkedIn to move massive amounts of data. At first, it was a highly technical undertaking, but over time the company has been trying to make it increasingly accessible to a greater number of people inside an organization.
In addition to Stream Designer, the company is also announcing Stream Governance, which helps ensure the data is being used correctly and only authorized users can see data.
“For a lot of organizations it’s almost like there’s two competing pressures: One is to unlock the data and use it to be effective and serve customers better and be more efficient. And the other is to lock it up and be safe and don’t let anything bad happen. And unless you have tooling that helps with both of these dimensions, you kind of end up a little bit stuck,” Kreps said. Stream Governance helps users make sure they are using the data flowing through these data streams in a safe and compliant way.
While they were at it, Confluent announced a new Confluent for Startups program, designed to get startups using the platform with free credits and access to expertise to help them get started with data streaming technologies.
Stream Designer and Stream Governance are available starting today as part of the Confluent cloud service subscription package, and there will be no additional charge for the new capabilities, he said.