The arrival of Apache Kafka 3.0 by Cloudera

[#1]

Introduction of Apache Kafka 3.0

Thanks to a collaborative effort between Cloudera and other parties, Apache Kafka continues to improve rapidly, and now, this has resulted in the release of Apache Kafka 3.0!

The top two questions now are:

• What does this mean for the setup that I have today?
• Why are we excited about Apache Kafka 3.0? Any challenges we should be aware of?

Let us address both of these.

What does this mean for the setup that I have today?

First of all, there is no need to be concerned about your current footprint.

Every time Cloudera announces a release, it also announces an end of support date. The availability of a new version does not detract from this. 

Furthermore, the Kafka 2 branch is very much still alive, and we are still planning to release improvements to this for a good period of time.

Why are we excited about Apache Kafka 3.0? Any challenges we should be aware of?

In short, we see Kafka 3.0 as a good foundation. There are many areas with small improvements, but we believe the following two will have the largest impact:

KRaft

Significant improvements have been made to KRaft, an initiative that aims to absorb the complexity of coordination into the Kafka project itself, rather than leveraging Apache ZooKeeper. This is mostly relevant for customers who do not use any other solution that leverages Zookeeper.

At the time of writing, KRaft (kicked off by KIP-500 and enriched by initiatives such as KIP-630) is not yet ready for production, but perhaps by the time you read this article, a production-ready version is available in a 3.x release.

Big Cleanup

Though not directly valuable to end-users, a large clean-up has been executed, resulting in the deprecation (not yet discontinuation) of old message formats V0 and V1, so perhaps already stop using these for new developments. Users are also discouraged from setting up Kafka 3 clusters on versions below Java 11. 

Conclusion

Kafka 3.0 offers a clean slate and a good foundation for enabling new developments. On the other hand the 3.0 version is clearly a .0 version and its main improvement is not ready for production usage, so the 2.x branch will stay in the spotlight a little longer. Cloudera will continue to keep developing and evaluating versions internally and as soon as a 3.x version becomes valuable for production, a Cloudera announcement will follow shortly!

For enthusiasts who want to know the details, the full list of improvements and changes can be found here.

Authored by

Dennis Jaheruddin (Global Streaming SME Lead) & Joseph Niemiec (Product Manager Kafka)