How to add hdfs property using cm rest api

[#1]

We need to add proxyuser properties to hdfs-site.xml during installation of our product.

CM REST API allows only update of existing property.

Does anyone know how to add property without GUI?

I tried the gollowing statement, but it fails:

curl -iv -X PUT -H "Content-Type:application/json" -H "Accept:application/json" -d '{"items":[{ "name": "ztsps_hadoop_proxyuser_groups_list","value": "*", "required": false, "default": "*", "displayName": "PS Proxy User Hosts", "description" : "PS user groups", "relatedName": "hadoop.proxyuser.ztsps.groups", "validationState" : "OK", "validationWarningsSuppressed" : false}]}' http://admin:admin@10.1.0.1:7180/api/v12/clusters/cluster/services/hdfs/config

It returns:

{
"message" : "Unknown configuration attribute 'ztsps_hadoop_proxyuser_groups_list'."
* Connection #0 to host 10.1.0.1 left intact
* Closing connection #0
}

Any help will be greatly appreciated,

Igor Kiselev.

[#2]

Any non-defined property would need to be added verbatim (in XML property tag form) to the Advanced Configuration Snippets. In your case you need to add the entry into the HDFS config fields named "Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml", i.e. CM name "core_site_safety_valve" [1].

[1] - http://www.cloudera.com/documentation/enterprise/latest/topics/cm_props_cdh570_hdfs.html

[#3]

Harsh,

thank you very much for your response. I looked at the doc and it instructs how to add property through GUI.

We are looking for command line capability. Does it exist? Anything, REST API or some script(Ambari has one)?

Please, let me know even if the only way is GUI. I would need to rethink our approach, if Cloudera does not provide command line capability to add properties.

I will really appreciate your answer, since it will spare me from fruitless research attemts and will save me time. 

Regards,

Igor Kiselev.

[#4]

Apologies if the post was ambiguous. I was speaking on the API lines, in that you need to add it in XML form into that quoted field name, rather than try to add a new property name that it won't accept (your original curl).

You require to do something like this (notice the name of the config being the safety valve described earlier, and that the value is in XML form with the contents of what you are trying to add in):

curl -iv -X PUT -H "Content-Type:application/json" -H "Accept:application/json" -d '{"items":[{ "name": "core_site_safety_valve","value": "<property><name>hadoop.proxyuser.ztsps.users</name><value>*</value></property><property><name>hadoop.proxyuser.ztsps.groups</name><value>*</value></property>"}]}' http://admin:admin@10.1.0.1:7180/api/v12/clusters/cluster/services/hdfs/config

[#5]

Harsh,

thank you very much. I was glad to find out after reading your response that solution that I came to right one while waiting for the response.

May I ask you another question. I also need to modify value of mapreduce.application.classpath property. It is in mapred-site.xml. I cannot find API ways to do this. I cannot even retrieve any mapred properties with API. There is no mapred service. Only yarn. Could you, please, help.

Or should I open another topic?

Regards,

Igor Kiselev.

[#6]

Always better to have one Q per topic, for easy searchability/context. But in any case, YARN service in CM also includes MR2 properties. The page at http://www.cloudera.com/documentation/enterprise/latest/topics/cm_props_cdh570_yarn_mr2included_.htm... lists all available CM configs for YARN+MR2 service.

You are looking for the config in CM named "mapreduce_application_classpath" with default value '$HADOOP_MAPRED_HOME/*, $HADOOP_MAPRED_HOME/lib/*, $MR2_CLASSPATH'?

[#7]

Harsh,

thank you again and yes, I am looking for mapreduce_application_classpath. I need to append the value. Unfortunately curl -X GET -i http://admin:admin@10.1.0.1:7180/api/v12/clusters/cluster/services/yarn/config?view=full does not return this property at all. And there is no mapred service.

How I can get and modify this value through API?

As usual, your answer will be greatly appreaciated. This looks like my last problem.

Regards,

Igor Kiselev.

[#8 · Accepted solution]

This is certainly interesting and I can reproduce it.

I do find that if I query the GATEWAY role directly it shows up the MR2 configs. Given that MR2 is a purely GATEWAY + JOBHISTORYSERVER aspect in YARN, this would make sense, but I agree the service-level exhaustive config ought to show it too, and I'll report this internally to see what's up and if it can be changed to show the values in the future.

Something such as the below would work in the meantime:

curl -k -iv -X GET 'http://admin:admin@HOST:7180/api/v12/clusters/cluster/services/yarn/roles/YARN-GATEWAY-1/config?view=full'

You can grab your gateway rolename from the services/yarn/roles endpoint first, in case its something custom. In my example above the rolename was "YARN-GATEWAY-1".

[#9]

Harsh,

thank you very much for all your help. I found how how to create and use roles through API.

Everything works.

Regards,

Igor Kiselev.

[#10]

In checking with CM Engg. internally, it appears I was mistaken in believing that the service/config endpoint was supposed to return *all* configs. Its only designed to emit service config, and the gateway configs which are separately maintained in each service should be gotten from the role group config endpoint instead. This is by design.