Created 12-28-2022 10:55 PM
What is Kafka Gateway and Kafka MirrorMaker when add Role Instances to Kafka?
I created a Kafka service on Cloudera Manager. Now I want to add a new Kafka broker instance inside the Kafka service. I followed the guide on the internet. Choose Instances -> Add Role Instances
A new window comes up. But I notice that I can only add Kafka Brokers Instances (this guide said that I can add Kafka Connect Instance too).
Also, there are two other instances called Gateway and Mirrormaker, which I don't know what are they? I search google but only find some info about Kafka MirrorMaker but no luck on finding anything about Kafka Gateway
Created 01-02-2023 10:03 PM
Hello @quangbilly79
Thanks for using Cloudera Community. Based on your Post, you may consider "Kafka Gateway" as the Client for Kafka, which are setup on the Hosts wherein the same is added as per Cloudera Manager "Assign Roles". A Client/Gateway is familiar with the Service (Kafka in this Case) & all Client/Service Configs are available for the Client/Gateway without any manual intervention. Any changes made to the Service or Client Configs is pushed to the Service/Client Configuration by Cloudera Manager.
Imagine a Scenario wherein you wish to run "hdfs dfs -ls" on a HDFS FileSystem. Simply running the Command won't work unless the Host wherein the Command "hdfs dfs -ls" is being run knows the Setup (HDFS FileSystem, NameNode, Port, Protocol). Review [1] for an Example. Adding an HDFS Gateway ensures User doesn't need to manually configure a Client/Gateway with Cloudera Manager doing the needful.
Similarly, Kafka Gateway operates. Else, Customer need to manually configure the Client/Gateway Setup. Hope the above answer your query concerning the Gateway Role.
Regards, Smarak
[1] https://www.ibm.com/docs/en/spectrum-scale-bda?topic=hdfs-clients-configuration
Created 01-02-2023 10:03 PM
Hello @quangbilly79
Thanks for using Cloudera Community. Based on your Post, you may consider "Kafka Gateway" as the Client for Kafka, which are setup on the Hosts wherein the same is added as per Cloudera Manager "Assign Roles". A Client/Gateway is familiar with the Service (Kafka in this Case) & all Client/Service Configs are available for the Client/Gateway without any manual intervention. Any changes made to the Service or Client Configs is pushed to the Service/Client Configuration by Cloudera Manager.
Imagine a Scenario wherein you wish to run "hdfs dfs -ls" on a HDFS FileSystem. Simply running the Command won't work unless the Host wherein the Command "hdfs dfs -ls" is being run knows the Setup (HDFS FileSystem, NameNode, Port, Protocol). Review [1] for an Example. Adding an HDFS Gateway ensures User doesn't need to manually configure a Client/Gateway with Cloudera Manager doing the needful.
Similarly, Kafka Gateway operates. Else, Customer need to manually configure the Client/Gateway Setup. Hope the above answer your query concerning the Gateway Role.
Regards, Smarak
[1] https://www.ibm.com/docs/en/spectrum-scale-bda?topic=hdfs-clients-configuration