Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to add hdfs property using cm rest api

avatar
Contributor

We need to add proxyuser properties to hdfs-site.xml during installation of our product.

CM REST API allows only update of existing property.

Does anyone know how to add property without GUI?

I tried the gollowing statement, but it fails:

curl -iv -X PUT -H "Content-Type:application/json" -H "Accept:application/json" -d '{"items":[{ "name": "ztsps_hadoop_proxyuser_groups_list","value": "*", "required": false, "default": "*", "displayName": "PS Proxy User Hosts", "description" : "PS user groups", "relatedName": "hadoop.proxyuser.ztsps.groups", "validationState" : "OK", "validationWarningsSuppressed" : false}]}' http://admin:admin@10.1.0.1:7180/api/v12/clusters/cluster/services/hdfs/config

 

It returns:

{
"message" : "Unknown configuration attribute 'ztsps_hadoop_proxyuser_groups_list'."
* Connection #0 to host 10.1.0.1 left intact
* Closing connection #0
}

 

Any help will be greatly appreciated,

Igor Kiselev.

1 ACCEPTED SOLUTION

avatar
Mentor
This is certainly interesting and I can reproduce it.

I do find that if I query the GATEWAY role directly it shows up the MR2 configs. Given that MR2 is a purely GATEWAY + JOBHISTORYSERVER aspect in YARN, this would make sense, but I agree the service-level exhaustive config ought to show it too, and I'll report this internally to see what's up and if it can be changed to show the values in the future.

Something such as the below would work in the meantime:

curl -k -iv -X GET 'http://admin:admin@HOST:7180/api/v12/clusters/cluster/services/yarn/roles/YARN-GATEWAY-1/config?view=full'

You can grab your gateway rolename from the services/yarn/roles endpoint first, in case its something custom. In my example above the rolename was "YARN-GATEWAY-1".

View solution in original post

14 REPLIES 14

avatar
New Contributor

Is there some place where the Cloudera  property / configuration grouping is documented for the API?

 

I am trying to set up a script using the RESTful API that can repeatedly, consistently configure a complete Cloudera cluster. For instance, given a cluster with only (say) ten hosts configured but no services (yet), how would I use the API to configure and set up the HDFS NameNode, SecondaryNameNode, DataNodes, Gateway, etc? I can figure out the parameters, but can't figure out how to assign roles.

 

thanks

avatar
Explorer

Hi Harsh, In my case, i am trying to fetch all mapreduce related properties. But in the yarn, there is no role of type Gateway for my cluster.  So is there any other way to fetch mapreduce properties?

Also if i try to fetch mapreduce-site.xm file from other roles like node manager or resourcemanager, it is just givivng the xml file with 2 properties only:

mapreduce.jobhistory.webapp.address
mapreduce.jobhistory.webapp.https.address
Could you please explain what could be the reason for that?

And how can i fetch all the properties.

avatar
Mentor
@priyanka2,

> But in the yarn, there is no role of type Gateway for my cluster.
> So is there any other way to fetch mapreduce properties?

There may still be a role config group for it. You can use the roleConfigGroups endpoint to access its configs:

Something like `curl -u auth:props -v http://cm-host.com:7180/api/v15/clusters/MyClusterName/services/YARN-1/roleConfigGroups/YARN-1-GATEW...`

> Could you please explain what could be the reason for that?

The NodeManagers do not require MR client-side properties, just properties related to services it may need to contact and the MR shuffle service plugin configs. The NM is not involved in the MR app-side framework execution, so its mapred-site.xml only carries a subset as you've observed.

@mikefisch,

IIUC, you are looking for a way to assign roles to specific hosts?

Use the POST call described here, for each service endpoint: https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_services_-serviceName-_ro... -- Specifically, the roles list needs a structure that also requires a host reference ID that you can grab from the cluster hosts endpoint prior to this step.

There's a simpler auto-assign feature also available: https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_autoAssignRoles.html

avatar
Explorer

@Harsh J  Thanks for the reply. Yes http://cm-host.com:7180/api/v15/clusters/MyClusterName/services/YARN-1/roleConfigGroups/YARN-1-GATEW... api is giving all the mapreduce properties.

So If i want to fetch all defined mapreduce properties,can i use this Api or it does have any pre-requisites?

By prerequisites i mean, as in my cluster there was no role of gateway type for YARN service, even then using above api is giving me all mapred properties.

Also does it require any privileges to access this api?

avatar
Mentor
> So If i want to fetch all defined mapreduce properties,can i use this Api
or it does have any pre-requisites?

Yes you can. The default role group mostly always exists even if role
instances do not, but if not (such as in a heavily API driven install) you
can create one before you fetch.

> Also does it require any privileges to access this api?

A read-only user should also be able to fetch configs as a GET call over
API. However, if there are configs marked as secured (such as configs that
carry passwords, etc.) then the value retrieval will require admin
privileges - they will otherwise appear redacted.