> So If i want to fetch all defined mapreduce properties,can i use this Api or it does have any pre-requisites? Yes you can. The default role group mostly always exists even if role instances do not, but if not (such as in a heavily API driven install) you can create one before you fetch. > Also does it require any privileges to access this api? A read-only user should also be able to fetch configs as a GET call over API. However, if there are configs marked as secured (such as configs that carry passwords, etc.) then the value retrieval will require admin privileges - they will otherwise appear redacted. ... View more

@priyanka2, > But in the yarn, there is no role of type Gateway for my cluster. > So is there any other way to fetch mapreduce properties? There may still be a role config group for it. You can use the roleConfigGroups endpoint to access its configs: Something like `curl -u auth:props -v http://cm-host.com:7180/api/v15/clusters/MyClusterName/services/YARN-1/roleConfigGroups/YARN-1-GATEWAY-BASE/config?view=full` > Could you please explain what could be the reason for that? The NodeManagers do not require MR client-side properties, just properties related to services it may need to contact and the MR shuffle service plugin configs. The NM is not involved in the MR app-side framework execution, so its mapred-site.xml only carries a subset as you've observed. @mikefisch, IIUC, you are looking for a way to assign roles to specific hosts? Use the POST call described here, for each service endpoint: https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_services_-serviceName-_roles.html -- Specifically, the roles list needs a structure that also requires a host reference ID that you can grab from the cluster hosts endpoint prior to this step. There's a simpler auto-assign feature also available: https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_autoAssignRoles.html ... View more

To add on: If you will not require audits or lineage at all for your cluster, you can also choose to disable their creation: Impala - Configuration - "Enable Impala Lineage Generation" (uncheck) Impala - Configuration - "Enable Impala Audit Event Generation" (uncheck) If you are using Navigator with Cloudera Enterprise, then these audits and lineage files should be sent automatically to the Navigator services. If they are not passing through, it may be an indicator of problem in the pipeline - please raise a support case if this is true. ... View more

For CDH / CDK Kafka users, the command is already in your PATH as "kafka-consumer-groups". ... View more

Could you share the full log from this failure, both from the Oozie server for the action ID and the action launcher job map task logs? The 8042 port is the NodeManager HTTP port, useful in serving logs of live containers among other status details over REST. It is not directly used by DistCp in its functions, but MapReduce and Oozie diagnostics might be invoking it as part of a response to a failure, so it is a secondary symptom. Note though that running DistCp via Oozie requires you to provide appropriate configs that ensures delegation tokens for both kerberized clusters are acquired. Use "mapreduce.job.hdfs-servers" with a value such as "hdfs://namenode-cluster-1,hdfs://namenode-cluster-2" to influence this on the Oozie server's delegation token acquisition phase. This is only relevant if you use Kerberos on both clusters. ... View more