API to manage YARN Capacity Queue

[#1]

Is there a Java or REST API for mapping users to capacity queues similar to what is accomplished by setting "yarn.scheduler.capacity.queue-mappings" in "capacity-scheduler.xml". Looking for a programatic approach to manage queues as new users are added to the system.

[#2]

@Tim Veil I believe the REST API for the scheduler only supports GET operations as per https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

[#3]

I started there but didn't see anything that looked like what i needed. There are a few options to PUT or POST but i didn't see anything that mapped users to queues. Lets see what else we can find.

[#4]

@Tim Veil

I could not find REST API to update queue mappings however I could think of using /var/lib/ambari-server/resources/scripts/configs.sh to update queue mapping(yarn.scheduler.capacity.queue-mappings) programmatically

[#5 · Accepted solution]

@Tim Veil It is possible to use the Ambari REST API to change that config. Below is an example:

curl -v -u admin:admin -H "Content-Type: application/json" -H "X-Requested-By:ambari" -X PUT http://<AMBARI-SERVER>:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANC... --data '{ "Clusters": { "desired_config": [ { "type": "capacity-scheduler", "tag": "version14534007568115", "service_config_version_note": "To test", "properties": { "yarn.scheduler.capacity.maximum-am-resource-percent": 0.2, "yarn.scheduler.capacity.maximum-applications": 10000, "yarn.scheduler.capacity.node-locality-delay": 40, "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator", "yarn.scheduler.capacity.queue-mappings-override.enable": false, "yarn.scheduler.capacity.root.acl_administer_queue": "*", "yarn.scheduler.capacity.root.capacity": 100, "yarn.scheduler.capacity.root.queues": "default", "yarn.scheduler.capacity.root.accessible-node-labels": "*", "yarn.scheduler.capacity.root.default.acl_submit_applications": "*", "yarn.scheduler.capacity.root.default.maximum-capacity": 100, "yarn.scheduler.capacity.root.default.user-limit-factor": 0.5, "yarn.scheduler.capacity.root.default.state": "RUNNING", "yarn.scheduler.capacity.root.default.capacity": 100 } } ] } }'

[#6]

@Ian Roberts I tried this but I'm getting the Exception below. Any ideas? Im able to hit the URL http://xxx:8080/api/v1/clusters/AT/ and get configuration output on a browser.

{"message":"RA040 I/O error while requesting Ambari","status":500,"trace":"org.apache.ambari.view.utils.ambari.AmbariApiException: RA040 I/O error while requesting Ambari org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:176) org.apache.ambari.view.capacityscheduler.ConfigurationService.writeConfiguration(ConfigurationService.java:409)...

Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://xxx:8080/api/v1/clusters/AT/ sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627) java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:209)\n\t... 97 more\n"}

[#7]

Hi @Ian Roberts,

I know this is a bit of an old thread but I was wondering if you know if the syntax has changed at all with the new HDP version. I have been using HDP 2.5 sandbox and calling this gives me a 415 error indicating that application/json requests are not supported. Do you know how I can implement this programatically in the new HDP version?

Thanks,

John

[#8]

I have figured out a solution. For anyone who is interested in this particular problem the approach that I took was to run the following commands:

curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d @<json-file> http://<ambari-server>:8080/api/v1/clusters/<cluster-name>;

where <json-file> has the same structure as mentioned in @Ian Roberts solution. Note that 'tag' needs to have a unique value. The resource manager in yarn then needs to be restarted which can be done via:

curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d '{"ServiceComponentInfo":{"state":"INSTALLED"}}' http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/YARN/components/RESOURCEMANAGER

curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d '{"ServiceComponentInfo":{"state":"STARTED"}}' http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/YARN/components/RESOURCEMANAGER

All the best,

John

[#9]

@Tim Veil

Please refer link for working command to achieve queue addition using script / in automated way -

https://community.hortonworks.com/questions/155903/how-to-add-new-yarn-queue-using-rest-api-ambari-c...