@quangbilly79 NO, you should not be adding gateway to every node.   This gateway should only be installed on the edge/utility nodes, where you give access to external systems and users.   These gateway nodes then are able to reach rest of the service(s) nodes. ... View more

Hi @quangbilly79    Cloudera will support YARN and Kubernets deployment mode and it will not support Standalone mode (In standalone mode you can access the Spark Master using 7077 port).   In order to check which node driver is launched and which node is executor is launched you need to go to Spark UI or Spark History Server UI of that application. From there go to Executors tab. You can see list of executors.   In the second table you find executor id. Where the executor id is 'driver' that is the one Driver Node and remaining all are executors. ... View more

Hello @quangbilly79    Thanks for using Cloudera Community. The "Spark Master" refers to the Resource Manager responsible for allocating resources. Since you are using YARN, Your Team needs to use "--master yarn". The usage of "--master spark://<IP Address>:7077" is for Spark Standalone Cluster, which isn't the Case for your team.   To your Observation concerning the "Driver Instance" & "Worker Instance" being added via "Add Role Instance", there is no such Option as YARN is the Resource Manager, which shall allocate the resources for Spark Driver & Executors.   Review [1] for the usage of "--master" as well. Hope the above answers your Team's queries.   Regards, Smarak   [1] https://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit ... View more

Hello @quangbilly79    Thanks for using Cloudera Community. Based on your Post, you may consider "Kafka Gateway" as the Client for Kafka, which are setup on the Hosts wherein the same is added as per Cloudera Manager "Assign Roles". A Client/Gateway is familiar with the Service (Kafka in this Case) & all Client/Service Configs are available for the Client/Gateway without any manual intervention. Any changes made to the Service or Client Configs is pushed to the Service/Client Configuration by Cloudera Manager.    Imagine a Scenario wherein you wish to run "hdfs dfs -ls" on a HDFS FileSystem. Simply running the Command won't work unless the Host wherein the Command "hdfs dfs -ls" is being run knows the Setup (HDFS FileSystem, NameNode, Port, Protocol). Review [1] for an Example. Adding an HDFS Gateway ensures User doesn't need to manually configure a Client/Gateway with Cloudera Manager doing the needful.    Similarly, Kafka Gateway operates. Else, Customer need to manually configure the Client/Gateway Setup. Hope the above answer your query concerning the Gateway Role.    Regards, Smarak   [1] https://www.ibm.com/docs/en/spectrum-scale-bda?topic=hdfs-clients-configuration ... View more