Dear CDH Users,
We are pleased to announce the release of the Cloudera Distribution of Apache Kafka 2.0 for CDH 5.
Apache Kafka is a highly scalable, distributed, publish-subscribe messaging system. This release is based on Apache Kafka 0.9, and adds security features such as Kerberos authentication, wire encryption, secure mirroring, a new consumer API, per-user throttling, and many other features and bug fixes that solidify Kafka as an enterprise production-grade component of the Hadoop ecosystem. Kafka 2.0 also ships with new management tooling in Cloudera Manager, for point-and-click configuration of each new capability.
New Features in Cloudera Distribution of Apache Kafka 2.0
- Kafka is rebased on Apache Kafka 0.9: http://archive.apache.org/dist/kafka/0.9.0.0/RELEASE_NOTES.html.
- Kerberos authentication of connections from clients and other brokers, including to ZooKeeper.
- Wire encryption of communications from clients and other brokers using SSL.
- A new client API for consumers (Java).
- A refactored, secure MirrorMaker to prevent data loss and improve reliability of cross-data center replication.
- Per-user quotas to throttle producer and consumer throughput in a multitenant cluster.
Requirements for Cloudera Distribution of Apache Kafka 2.0
- Cloudera Manager 5.5.3
- Any CDH 5.x release is supported.
Notable Issues Fixed in Cloudera Distribution of Apache Kafka 2.0
- Notable fixes backported into Kafka 2.0:
- KAFKA-2799: WakupException thrown in the followup poll() could lead to data loss
- KAFKA-2942: Inadvertent auto-commit when pre-fetching can cause message loss
- KAFKA-2878: Kafka broker throws OutOfMemory exception with invalid join group request
- KAFKA-2882: Add constructor cache for Snappy and LZ4 Output/Input stream in Compressor.java
- KAFKA-2913: GroupMetadataManager unloads all groups in removeGroupsForPartitions
- KAFKA-2880: Fetcher.getTopicMetadata NullPointerException when broker cannot be reached
- KAFKA-2950: Fix performance regression in the producer
- KAFKA-2973: Fix leak of child sensors on remove
- KAFKA-2978: Consumer stops fetching when consumed and fetch positions get out of sync
- KAFKA-2988: Change default configuration of the log cleaner
- KAFKA-3012: Avoid reserved.broker.max.id collisions on upgrade
- All backported fixes can be viewed in the git release notes here.
We look forward to you trying Kafka 2.0! , For more information, please use the links below:
As always, we welcome your feedback. Please send your comments and suggestions through our community forums.
