How to add hdfs property using cm rest api

[#1]

We need to add proxyuser properties to hdfs-site.xml during installation of our product.

CM REST API allows only update of existing property.

Does anyone know how to add property without GUI?

I tried the gollowing statement, but it fails:

curl -iv -X PUT -H "Content-Type:application/json" -H "Accept:application/json" -d '{"items":[{ "name": "ztsps_hadoop_proxyuser_groups_list","value": "*", "required": false, "default": "*", "displayName": "PS Proxy User Hosts", "description" : "PS user groups", "relatedName": "hadoop.proxyuser.ztsps.groups", "validationState" : "OK", "validationWarningsSuppressed" : false}]}' http://admin:admin@10.1.0.1:7180/api/v12/clusters/cluster/services/hdfs/config

It returns:

{
"message" : "Unknown configuration attribute 'ztsps_hadoop_proxyuser_groups_list'."
* Connection #0 to host 10.1.0.1 left intact
* Closing connection #0
}

Any help will be greatly appreciated,

Igor Kiselev.

[#2]

Is there some place where the Cloudera  property / configuration grouping is documented for the API?

I am trying to set up a script using the RESTful API that can repeatedly, consistently configure a complete Cloudera cluster. For instance, given a cluster with only (say) ten hosts configured but no services (yet), how would I use the API to configure and set up the HDFS NameNode, SecondaryNameNode, DataNodes, Gateway, etc? I can figure out the parameters, but can't figure out how to assign roles.

thanks

[#3]

Hi Harsh, In my case, i am trying to fetch all mapreduce related properties. But in the yarn, there is no role of type Gateway for my cluster.  So is there any other way to fetch mapreduce properties?

Also if i try to fetch mapreduce-site.xm file from other roles like node manager or resourcemanager, it is just givivng the xml file with 2 properties only:

mapreduce.jobhistory.webapp.address
mapreduce.jobhistory.webapp.https.address
Could you please explain what could be the reason for that?

And how can i fetch all the properties.

[#4]

@priyanka2,

> But in the yarn, there is no role of type Gateway for my cluster.
> So is there any other way to fetch mapreduce properties?

There may still be a role config group for it. You can use the roleConfigGroups endpoint to access its configs:

Something like `curl -u auth:props -v http://cm-host.com:7180/api/v15/clusters/MyClusterName/services/YARN-1/roleConfigGroups/YARN-1-GATEW...`

> Could you please explain what could be the reason for that?

The NodeManagers do not require MR client-side properties, just properties related to services it may need to contact and the MR shuffle service plugin configs. The NM is not involved in the MR app-side framework execution, so its mapred-site.xml only carries a subset as you've observed.

@mikefisch,

IIUC, you are looking for a way to assign roles to specific hosts?

Use the POST call described here, for each service endpoint: https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_services_-serviceName-_ro... -- Specifically, the roles list needs a structure that also requires a host reference ID that you can grab from the cluster hosts endpoint prior to this step.

There's a simpler auto-assign feature also available: https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_autoAssignRoles.html

[#5]

@Harsh J  Thanks for the reply. Yes http://cm-host.com:7180/api/v15/clusters/MyClusterName/services/YARN-1/roleConfigGroups/YARN-1-GATEW... api is giving all the mapreduce properties.

So If i want to fetch all defined mapreduce properties,can i use this Api or it does have any pre-requisites?

By prerequisites i mean, as in my cluster there was no role of gateway type for YARN service, even then using above api is giving me all mapred properties.

Also does it require any privileges to access this api?

[#6]

> So If i want to fetch all defined mapreduce properties,can i use this Api
or it does have any pre-requisites?

Yes you can. The default role group mostly always exists even if role
instances do not, but if not (such as in a heavily API driven install) you
can create one before you fetch.

> Also does it require any privileges to access this api?

A read-only user should also be able to fetch configs as a GET call over
API. However, if there are configs marked as secured (such as configs that
carry passwords, etc.) then the value retrieval will require admin
privileges - they will otherwise appear redacted.