The following versions of Apache Kafka have been incorporated in HDP 2.2.0 to 2.6.1: 0.8.1, 0.8.2, 0.9.0, 0.10.0, 0.10.1. Apache Kafka is now at 0.11.
Hortonworks is working to make Kafka easier for enterprises to use. New focus areas include creation of a Kafka Admin Panel to create/delete topics and manage user permissions, easier and safer distribution of security tokens and support for multiple ways of publishing/consuming data via a Kafka REST server/API. Here are a few areas of strong contribution:
Rack awareness for Increased resilience and availability such that replicas are isolated so they are guaranteed to span multiple racks or availability zones.
Automated replica leader election for automated, even distribution of leaders in a cluster capability by detecting uneven distribution with some brokers serving more data compared to others and makes adjustments.
Message Timestamps so every message in Kafka now has a timestamp field that indicates the time at which the message was produced.
SASL improvements including external authentication servers and support of multiple types of SASL authentication on one server
Ambari Views for visualization of Kafka operational metrics
Kafka security encompasses multiple needs – the need to encrypt the data flowing through Kafka and preventing rogue agents from publishing data to Kafka, as well as the ability to manage access to specific topics on an individual or group level.
As a result, latest updates in Kafka support wire encryption via SSL, Kerberos based authentication and granular authorization options via Apache Ranger or other pluggable authorization system.
This article lists below new features beyond Hortonworks contribution. At the high level, the following have been added by the overall community.
Kafka Streams API
Kafka Connect API
New unified Consumer API
Transport encryption using TLS/SSL
Kerberos/SASL Authentication support
Access Control Lists
Timestamps on messages
Reduced client dependence on zookeeper (offsets stored in Kafka topic)
New Features Since HDP 2.2
Here is the list of NEW FEATURES as they have been included in the release notes.