Product Announcements
Find the latest product announcements and version updates

[ANNOUNCE] Cloudera Distribution of Apache Kafka 2.0 for CDH 5 Released

Master Collaborator

Dear CDH Users,


We are pleased to announce the release of the Cloudera Distribution of Apache Kafka 2.0 for CDH 5.  


Apache Kafka is a highly scalable, distributed, publish-subscribe messaging system. This release is based on Apache Kafka 0.9, and adds security features such as Kerberos authentication, wire encryption, secure mirroring, a new consumer API, per-user throttling, and many other features and bug fixes that solidify Kafka as an enterprise production-grade component of the Hadoop ecosystem. Kafka 2.0 also ships with new management tooling in Cloudera Manager, for point-and-click configuration of each new capability.


New Features in Cloudera Distribution of Apache Kafka 2.0

  • Kafka is rebased on Apache Kafka 0.9:
  • Kerberos authentication of connections from clients and other brokers, including to ZooKeeper.
  • Wire encryption of communications from clients and other brokers using SSL.
  • A new client API for consumers (Java).
  • A refactored, secure MirrorMaker to prevent data loss and improve reliability of cross-data center replication.
  • Per-user quotas to throttle producer and consumer throughput in a multitenant cluster.

Requirements for Cloudera Distribution of Apache Kafka 2.0


  • Cloudera Manager 5.5.3
  • Any CDH 5.x release is supported.


Notable Issues Fixed in Cloudera Distribution of Apache Kafka 2.0

  • Notable fixes backported into Kafka 2.0:
    • KAFKA-2799: WakupException thrown in the followup poll() could lead to data loss
    • KAFKA-2942: Inadvertent auto-commit when pre-fetching can cause message loss
    • KAFKA-2878: Kafka broker throws OutOfMemory exception with invalid join group request
    • KAFKA-2882: Add constructor cache for Snappy and LZ4 Output/Input stream in
    • KAFKA-2913: GroupMetadataManager unloads all groups in removeGroupsForPartitions
    • KAFKA-2880: Fetcher.getTopicMetadata NullPointerException when broker cannot be reached
    • KAFKA-2950: Fix performance regression in the producer
    • KAFKA-2973: Fix leak of child sensors on remove
    • KAFKA-2978: Consumer stops fetching when consumed and fetch positions get out of sync
    • KAFKA-2988: Change default configuration of the log cleaner
    • KAFKA-3012: Avoid collisions on upgrade
  • All backported fixes can be viewed in the git release notes here.

We look forward to you trying Kafka 2.0! , For more information, please use the links below:

As always, we welcome your feedback. Please send your comments and suggestions through our community forums.