Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to add hdfs property using cm rest api

SOLVED Go to solution

Configure a cluster with the REST API (was: How to add hdfs property using cm rest api)

New Contributor

Is there some place where the Cloudera  property / configuration grouping is documented for the API?

 

I am trying to set up a script using the RESTful API that can repeatedly, consistently configure a complete Cloudera cluster. For instance, given a cluster with only (say) ten hosts configured but no services (yet), how would I use the API to configure and set up the HDFS NameNode, SecondaryNameNode, DataNodes, Gateway, etc? I can figure out the parameters, but can't figure out how to assign roles.

 

thanks

Re: How to fetch mapreduce properties using cm rest api

Explorer

Hi Harsh, In my case, i am trying to fetch all mapreduce related properties. But in the yarn, there is no role of type Gateway for my cluster.  So is there any other way to fetch mapreduce properties?

Also if i try to fetch mapreduce-site.xm file from other roles like node manager or resourcemanager, it is just givivng the xml file with 2 properties only:

mapreduce.jobhistory.webapp.address
mapreduce.jobhistory.webapp.https.address
Could you please explain what could be the reason for that?

And how can i fetch all the properties.

Re: How to fetch mapreduce properties using cm rest api

Master Guru
@priyanka2,

> But in the yarn, there is no role of type Gateway for my cluster.
> So is there any other way to fetch mapreduce properties?

There may still be a role config group for it. You can use the roleConfigGroups endpoint to access its configs:

Something like `curl -u auth:props -v http://cm-host.com:7180/api/v15/clusters/MyClusterName/services/YARN-1/roleConfigGroups/YARN-1-GATEW...`

> Could you please explain what could be the reason for that?

The NodeManagers do not require MR client-side properties, just properties related to services it may need to contact and the MR shuffle service plugin configs. The NM is not involved in the MR app-side framework execution, so its mapred-site.xml only carries a subset as you've observed.

@mikefisch,

IIUC, you are looking for a way to assign roles to specific hosts?

Use the POST call described here, for each service endpoint: https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_services_-serviceName-_ro... -- Specifically, the roles list needs a structure that also requires a host reference ID that you can grab from the cluster hosts endpoint prior to this step.

There's a simpler auto-assign feature also available: https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_autoAssignRoles.html

Re: How to fetch mapreduce properties using cm rest api

Explorer

@Harsh J  Thanks for the reply. Yes http://cm-host.com:7180/api/v15/clusters/MyClusterName/services/YARN-1/roleConfigGroups/YARN-1-GATEW... api is giving all the mapreduce properties.

So If i want to fetch all defined mapreduce properties,can i use this Api or it does have any pre-requisites?

By prerequisites i mean, as in my cluster there was no role of gateway type for YARN service, even then using above api is giving me all mapred properties.

Also does it require any privileges to access this api?

Re: How to fetch mapreduce properties using cm rest api

Master Guru
> So If i want to fetch all defined mapreduce properties,can i use this Api
or it does have any pre-requisites?

Yes you can. The default role group mostly always exists even if role
instances do not, but if not (such as in a heavily API driven install) you
can create one before you fetch.

> Also does it require any privileges to access this api?

A read-only user should also be able to fetch configs as a GET call over
API. However, if there are configs marked as secured (such as configs that
carry passwords, etc.) then the value retrieval will require admin
privileges - they will otherwise appear redacted.