Confluent achieves Holy Grail of ‘exactly once’ delivery on Kafka messaging service

Confluent, the commercial company behind the open source messaging tool, Apache Kafka, announced a new feature this week that should please Kafka users — the ability to send a message using Kafka “exactly once.”

That may not sound like a big deal to laypeople, but to folks who use high speed messaging tools like Kafka, it is in fact a Holy Grail of sorts, something that people in the community believed they could never achieve.

Typically when you send a message, you wait for a response that the message was delivered, but as Confluent CTO Neha Narkhede explained, in a distributed messaging system like Kafka, sometimes stuff happens along the way. Computers fail, networks have hiccups — but when you’re dealing with say a financial transaction, you want to be sure that happens, well, exactly once and no more.

Most people believed that “exactly once” was an unattainable goal, that it would come at too high a trade-off in terms of delivery speed versus the ability to do this, but Narkhede says, the company put lots of engineering resources on the problem and after a year of tinkering have finally come up with a way to achieve this long-sought feature.

There are lots of technical details making this happen, and Narkhede admits there are technical trade-offs along the way, but not as many as you might think. In fact, the company has not only solved this issue, it has done so without sacrificing message speed, she said.

“If you look at performance overhead [using Exactly Once] is almost negligible, and the performance not using the guarantee [has gotten] better,” she said.

The new release is 20 percent faster across the board, with a 3-10 percent speed penalty for using the exactly once functionality. While she said there is always going to be some overhead in terms of speed for using exactly once, the company plans to cut down on that as much as possible in the coming months.

She admits there is a lot of skepticism in the community about this feature. “It’s kind of insane. It’s been an open problem for so many years and Kafka has solved it — but how do we know it actually works?” she asked echoing the doubts of the community.

“We ran thousands of hours of tests. We took a lot of care and focus on performance. How can we fundamentally rethink how Kafka was built and how can we make it all faster. A year down the line, [it’s ready] to put to use,” she told TechCrunch.

Confluent raised a $50 million funding round in March and has raised $80 million to-date. Kafka was originally created at LinkedIn and moved into the open source community. The founders formed Confluent in 2014.