With the launch of CDP Public Cloud 7.2.15, Cloudera Streams Messaging for Data Hub deployments has gotten some powerful new features! Streams Messaging now supports Multi-Availability Zone Deployments enabling High Availability, OAuth2 support for Clients connecting to Kafka Brokers and Schema Registry, Streams Messaging Manager Connect UI Changes, Kafka Connect security features, Debezium CDC Connectors, ability to import Kafka data to Atlas and much more.
Streams Messaging High Availability cluster definition and template.
You can use the template and definitions to deploy highly available Streams Messaging clusters that leverage multiple availability zones and ensure that functionality is not degraded when a single availability zone has an outage
Three new cluster definitions are introduced for Streams Messaging. The new definitions are as follows:
Streams Messaging High Availability for AWS
Streams Messaging High Availability for Azure (Technical Preview)
Streams Messaging High Availability for Google Cloud (Technical Preview)
OAuth2 authentication available for Kafka
Oauth2 authentication support is added for the Kafka service. You can now configure Kafka brokers to authenticate clients using Oauth2. For more information, see OAuth2 authentication. Changes for Streams Messaging Manager Kafka Connect UI The Streams Messaging Manager for deploying Kafka Connect Connectors has changed from a String based JSON object to a submission form. Other enhancements include:
Ability to import existing JSON Configurations and populate the form
Users can import NiFi Flow Definitions for Stateless Connectors and Enhance the forum with the Parameters defined in the NiFi Flow Definition
Mark forum fields as secret to protect specific configurations by storing them in the new Kafka Connect Secret Storage
Validation Messages now are shown at the specific field with the issue rather then an error message about the entire JSON configuration being used
Secure Kafka Connect Kafka Connect is now generally available in the Public Cloud and can be used in production environments. This is the result of multiple changes, improvements, and new features related to Kafka Connect security including the following:
Ranger support for Kafka Connect that allows policies to be defined at the Connect cluster level or the named connectors themselves allowing for secure multi-tenant experiences
Secret storage to securely store sensitive configurations used with connectors such as passwords or tokens
Connect REST API can be secured by enabling SPNEGO authentication.
Kafka Connect Connectors can be configured to override the JAAS, and restrict the usage of the Worker principal
Debezium Connector support
The following change data capture (CDC) connectors are added to Kafka Connect:
Debezium MySQL Source
Debezium Postgres Source
Debezium SQL Server Source
Debezium Oracle Source
Each of the connectors require CDP specific steps before they can be deployed. For more information, see Connectors.
Importing Kafka entities into Atlas
Kafka topics and clients can now be imported into Atlas as entities (metadata) using a new action available for the Kafka service in Cloudera Manager. The new action is available at Kafka service>Actions>Import Kafka Topics Into Atlas. The action serves as a replacement/alternative for the kafka-import.sh tool. For more information, see Importing Kafka entities into Atlas.