Cloudera’s Data In Motion Team is pleased to announce the release of the Cloudera Streaming Messaging Operator 1.1, an integral component of Cloudera Streaming - Kubernetes Operator. With this release, customers receive Kafka Connect support and Kafka replication in the operator.
Use Cases
- Loading and unloading data from Kafka: Kafka Connect gives Kafka users a simple way to get data quickly from a source and feed it to a Kafka topic. It also allows them to get data from a topic and copy it to an external destination. Adding Kafka Connect support to the operator gives our customers a tool for moving data in and out of Kafka, efficiently.
- Replicating data to other sites: Disaster resilience is an important aspect of any Kafka production deployment. The Cloudera Streaming Kubernetes Operator now supports configuring and running Kafka replication flows across any two Kafka clusters. These clusters could be in the same or in different data centers to provide increased resilience against disasters.
- Kafka migrations: Customers can migrate or replicate data between containerized Kafka clusters and on-prem or cloud-based clusters with the Cloudera Streaming Kubernetes Operator. Data can now be replicated in any direction and between two or more clusters at a time.
Release Highlights
- Rebase on Strimzi 0.41.0: This release of Cloudera Streaming Messaging Operator has been rebased on Strimzi 0.41.0.
- Kafka Connect support: Deploy Kafka Connect clusters and Kafka connectors using KafkaConnect and KafkaConnector resources.
- Kafka replication support: Set up data replication between Kafka clusters using Cloudera Streams Messaging Operator. This allows users to:
- The operator uses a Kafka Connect-based approach for replication of Kafka data that is scalable, robust, and fault tolerant. For example, it supports the same key features as MirrorMaker 2.
- Replication of Kafka topic partitions to have multiple copies of the same data in different Kafka clusters to avoid data loss in case of data center failure.
- Replication of Kafka consumer group offsets to be able to failover between clusters without losing data.
- Ability to monitor your replication at any time.
- In addition, Kafka Connect-based replication has a number of advantages over MirrorMaker 2. These include:
- Single Messages Transforms (SMTs) can be configured for data replication.
- Manipulating source offsets is possible using the Kafka Connect REST API.
- Some replication architectures, like unidirectional replication, require less resources and Kafka Connect groups when using overrides for heartbeating
- For more information, see Replication Overview.
For the complete list of fixes and improvements read these Release Notes .
Getting to the new release
To upgrade to Cloudera Stream Messaging Operator 1.1, check out this upgrade guide. Please note, if you are installing the operator for the first time use this installation overview.
Public Resources
