How to add hdfs property using cm rest api

[#1]

We need to add proxyuser properties to hdfs-site.xml during installation of our product.

CM REST API allows only update of existing property.

Does anyone know how to add property without GUI?

I tried the gollowing statement, but it fails:

curl -iv -X PUT -H "Content-Type:application/json" -H "Accept:application/json" -d '{"items":[{ "name": "ztsps_hadoop_proxyuser_groups_list","value": "*", "required": false, "default": "*", "displayName": "PS Proxy User Hosts", "description" : "PS user groups", "relatedName": "hadoop.proxyuser.ztsps.groups", "validationState" : "OK", "validationWarningsSuppressed" : false}]}' http://admin:admin@10.1.0.1:7180/api/v12/clusters/cluster/services/hdfs/config

It returns:

{
"message" : "Unknown configuration attribute 'ztsps_hadoop_proxyuser_groups_list'."
* Connection #0 to host 10.1.0.1 left intact
* Closing connection #0
}

Any help will be greatly appreciated,

Igor Kiselev.

[#2]

Any non-defined property would need to be added verbatim (in XML property tag form) to the Advanced Configuration Snippets. In your case you need to add the entry into the HDFS config fields named "Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml", i.e. CM name "core_site_safety_valve" [1].

[1] - http://www.cloudera.com/documentation/enterprise/latest/topics/cm_props_cdh570_hdfs.html

[#3]

Harsh,

thank you very much for your response. I looked at the doc and it instructs how to add property through GUI.

We are looking for command line capability. Does it exist? Anything, REST API or some script(Ambari has one)?

Please, let me know even if the only way is GUI. I would need to rethink our approach, if Cloudera does not provide command line capability to add properties.

I will really appreciate your answer, since it will spare me from fruitless research attemts and will save me time. 

Regards,

Igor Kiselev.

[#4]

Apologies if the post was ambiguous. I was speaking on the API lines, in that you need to add it in XML form into that quoted field name, rather than try to add a new property name that it won't accept (your original curl).

You require to do something like this (notice the name of the config being the safety valve described earlier, and that the value is in XML form with the contents of what you are trying to add in):

curl -iv -X PUT -H "Content-Type:application/json" -H "Accept:application/json" -d '{"items":[{ "name": "core_site_safety_valve","value": "<property><name>hadoop.proxyuser.ztsps.users</name><value>*</value></property><property><name>hadoop.proxyuser.ztsps.groups</name><value>*</value></property>"}]}' http://admin:admin@10.1.0.1:7180/api/v12/clusters/cluster/services/hdfs/config

[#5]

Harsh,

thank you very much. I was glad to find out after reading your response that solution that I came to right one while waiting for the response.

May I ask you another question. I also need to modify value of mapreduce.application.classpath property. It is in mapred-site.xml. I cannot find API ways to do this. I cannot even retrieve any mapred properties with API. There is no mapred service. Only yarn. Could you, please, help.

Or should I open another topic?

Regards,

Igor Kiselev.

[#6]

Always better to have one Q per topic, for easy searchability/context. But in any case, YARN service in CM also includes MR2 properties. The page at http://www.cloudera.com/documentation/enterprise/latest/topics/cm_props_cdh570_yarn_mr2included_.htm... lists all available CM configs for YARN+MR2 service.

You are looking for the config in CM named "mapreduce_application_classpath" with default value '$HADOOP_MAPRED_HOME/*, $HADOOP_MAPRED_HOME/lib/*, $MR2_CLASSPATH'?

[#7]

Harsh,

thank you again and yes, I am looking for mapreduce_application_classpath. I need to append the value. Unfortunately curl -X GET -i http://admin:admin@10.1.0.1:7180/api/v12/clusters/cluster/services/yarn/config?view=full does not return this property at all. And there is no mapred service.

How I can get and modify this value through API?

As usual, your answer will be greatly appreaciated. This looks like my last problem.

Regards,

Igor Kiselev.

[#8 · Accepted solution]

This is certainly interesting and I can reproduce it.

I do find that if I query the GATEWAY role directly it shows up the MR2 configs. Given that MR2 is a purely GATEWAY + JOBHISTORYSERVER aspect in YARN, this would make sense, but I agree the service-level exhaustive config ought to show it too, and I'll report this internally to see what's up and if it can be changed to show the values in the future.

Something such as the below would work in the meantime:

curl -k -iv -X GET 'http://admin:admin@HOST:7180/api/v12/clusters/cluster/services/yarn/roles/YARN-GATEWAY-1/config?view=full'

You can grab your gateway rolename from the services/yarn/roles endpoint first, in case its something custom. In my example above the rolename was "YARN-GATEWAY-1".

[#9]

Harsh,

thank you very much for all your help. I found how how to create and use roles through API.

Everything works.

Regards,

Igor Kiselev.

[#10]

In checking with CM Engg. internally, it appears I was mistaken in believing that the service/config endpoint was supposed to return *all* configs. Its only designed to emit service config, and the gateway configs which are separately maintained in each service should be gotten from the role group config endpoint instead. This is by design.

[#11]

Is there some place where the Cloudera  property / configuration grouping is documented for the API?

I am trying to set up a script using the RESTful API that can repeatedly, consistently configure a complete Cloudera cluster. For instance, given a cluster with only (say) ten hosts configured but no services (yet), how would I use the API to configure and set up the HDFS NameNode, SecondaryNameNode, DataNodes, Gateway, etc? I can figure out the parameters, but can't figure out how to assign roles.

thanks

[#12]

Hi Harsh, In my case, i am trying to fetch all mapreduce related properties. But in the yarn, there is no role of type Gateway for my cluster.  So is there any other way to fetch mapreduce properties?

Also if i try to fetch mapreduce-site.xm file from other roles like node manager or resourcemanager, it is just givivng the xml file with 2 properties only:

mapreduce.jobhistory.webapp.address
mapreduce.jobhistory.webapp.https.address
Could you please explain what could be the reason for that?

And how can i fetch all the properties.

[#13]

@priyanka2,

> But in the yarn, there is no role of type Gateway for my cluster.
> So is there any other way to fetch mapreduce properties?

There may still be a role config group for it. You can use the roleConfigGroups endpoint to access its configs:

Something like `curl -u auth:props -v http://cm-host.com:7180/api/v15/clusters/MyClusterName/services/YARN-1/roleConfigGroups/YARN-1-GATEW...`

> Could you please explain what could be the reason for that?

The NodeManagers do not require MR client-side properties, just properties related to services it may need to contact and the MR shuffle service plugin configs. The NM is not involved in the MR app-side framework execution, so its mapred-site.xml only carries a subset as you've observed.

@mikefisch,

IIUC, you are looking for a way to assign roles to specific hosts?

Use the POST call described here, for each service endpoint: https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_services_-serviceName-_ro... -- Specifically, the roles list needs a structure that also requires a host reference ID that you can grab from the cluster hosts endpoint prior to this step.

There's a simpler auto-assign feature also available: https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_autoAssignRoles.html

[#14]

@Harsh J  Thanks for the reply. Yes http://cm-host.com:7180/api/v15/clusters/MyClusterName/services/YARN-1/roleConfigGroups/YARN-1-GATEW... api is giving all the mapreduce properties.

So If i want to fetch all defined mapreduce properties,can i use this Api or it does have any pre-requisites?

By prerequisites i mean, as in my cluster there was no role of gateway type for YARN service, even then using above api is giving me all mapred properties.

Also does it require any privileges to access this api?

[#15]

> So If i want to fetch all defined mapreduce properties,can i use this Api
or it does have any pre-requisites?

Yes you can. The default role group mostly always exists even if role
instances do not, but if not (such as in a heavily API driven install) you
can create one before you fetch.

> Also does it require any privileges to access this api?

A read-only user should also be able to fetch configs as a GET call over
API. However, if there are configs marked as secured (such as configs that
carry passwords, etc.) then the value retrieval will require admin
privileges - they will otherwise appear redacted.