As Apache developers awoke one morning from uneasy dreams, they found themselves transformed in their beds into a giant new version of Apache Kafka. Version 0.9, released yesterday, adds security support for SSL for the first time, as well as new consumer APIs.
This latest version of Kafka includes updates that should make it easier to integrate this high-throughput distributed messaging systems into data-driven architectures. This release is the first to include the newly rewritten consumer client, which allows developers to use the deeper features of Kafka without having to use Java.
(Related: News about another Apache project: Ignite)
The Kafka consumer client is also much thinner now, allowing for fewer dependencies and less overhead. Another change to Kafka’s consumer client API allows multiple Kafka nodes to better run as a group thanks to the addition of a highly available coordinator node to the herd.
The new coordinator node will be able to make better consensus decisions about Apache ZooKeeper’s status and about when rebalancing operations complete successfully. Previously, systems using ZooKeeper for cluster control would sometimes see rebalancing failure in their Kafka clusters due to uncoordinated views of the server information kept in ZooKeeper.
Neha Narkhede, cofounder and head of engineering at Confluent (and one of the co-creators of Kafka), said that this release of the project includes the culmination of a great deal of work aimed at getting Kafka enterprise-ready. To that end, she said, this is the first release of Kafka to support permissions and authentication for users. This is also the first version to allow specific clients to be rate limited, enabling multi-tenancy support.
“Kafka has been adopted very widely across financial services,” said Narkhede. “This is a big step forward in making Kafka enterprise-ready.” She added that the new SSL support enables further security while data is on the wire.
Narkhede said that Kafka’s popularity is growing fast. In the first six months of 2015, the number of downloads grew 7x over. Meanwhile, she stated that job postings including Kafka as a desired skill have also grown by 400%.
Kafka has also won over some very heavy hitters. “The top two users of Apache Kafka are LinkedIn and Microsoft,” she said. “They both crossed a big milestone of serving 1 trillion messages a day using Kafka. Microsoft also announced that Bing and Office all use Kafka.”