Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

API to manage YARN Capacity Queue

Explorer

Is there a Java or REST API for mapping users to capacity queues similar to what is accomplished by setting "yarn.scheduler.capacity.queue-mappings" in "capacity-scheduler.xml". Looking for a programatic approach to manage queues as new users are added to the system.

1 ACCEPTED SOLUTION

Rising Star

@Tim Veil It is possible to use the Ambari REST API to change that config. Below is an example:

curl -v -u admin:admin -H "Content-Type: application/json" -H "X-Requested-By:ambari" -X PUT http://<AMBARI-SERVER>:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANC... --data '{ "Clusters": { "desired_config": [ { "type": "capacity-scheduler", "tag": "version14534007568115", "service_config_version_note": "To test", "properties": { "yarn.scheduler.capacity.maximum-am-resource-percent": 0.2, "yarn.scheduler.capacity.maximum-applications": 10000, "yarn.scheduler.capacity.node-locality-delay": 40, "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator", "yarn.scheduler.capacity.queue-mappings-override.enable": false, "yarn.scheduler.capacity.root.acl_administer_queue": "*", "yarn.scheduler.capacity.root.capacity": 100, "yarn.scheduler.capacity.root.queues": "default", "yarn.scheduler.capacity.root.accessible-node-labels": "*", "yarn.scheduler.capacity.root.default.acl_submit_applications": "*", "yarn.scheduler.capacity.root.default.maximum-capacity": 100, "yarn.scheduler.capacity.root.default.user-limit-factor": 0.5, "yarn.scheduler.capacity.root.default.state": "RUNNING", "yarn.scheduler.capacity.root.default.capacity": 100 } } ] } }'

View solution in original post

8 REPLIES 8

Rising Star

@Tim Veil I believe the REST API for the scheduler only supports GET operations as per https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Explorer

I started there but didn't see anything that looked like what i needed. There are a few options to PUT or POST but i didn't see anything that mapped users to queues. Lets see what else we can find.

Super Guru
@Tim Veil

I could not find REST API to update queue mappings however I could think of using /var/lib/ambari-server/resources/scripts/configs.sh to update queue mapping(yarn.scheduler.capacity.queue-mappings) programmatically

Rising Star

@Tim Veil It is possible to use the Ambari REST API to change that config. Below is an example:

curl -v -u admin:admin -H "Content-Type: application/json" -H "X-Requested-By:ambari" -X PUT http://<AMBARI-SERVER>:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANC... --data '{ "Clusters": { "desired_config": [ { "type": "capacity-scheduler", "tag": "version14534007568115", "service_config_version_note": "To test", "properties": { "yarn.scheduler.capacity.maximum-am-resource-percent": 0.2, "yarn.scheduler.capacity.maximum-applications": 10000, "yarn.scheduler.capacity.node-locality-delay": 40, "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator", "yarn.scheduler.capacity.queue-mappings-override.enable": false, "yarn.scheduler.capacity.root.acl_administer_queue": "*", "yarn.scheduler.capacity.root.capacity": 100, "yarn.scheduler.capacity.root.queues": "default", "yarn.scheduler.capacity.root.accessible-node-labels": "*", "yarn.scheduler.capacity.root.default.acl_submit_applications": "*", "yarn.scheduler.capacity.root.default.maximum-capacity": 100, "yarn.scheduler.capacity.root.default.user-limit-factor": 0.5, "yarn.scheduler.capacity.root.default.state": "RUNNING", "yarn.scheduler.capacity.root.default.capacity": 100 } } ] } }'

New Contributor

@Ian Roberts I tried this but I'm getting the Exception below. Any ideas? Im able to hit the URL http://xxx:8080/api/v1/clusters/AT/ and get configuration output on a browser.

{"message":"RA040 I/O error while requesting Ambari","status":500,"trace":"org.apache.ambari.view.utils.ambari.AmbariApiException: RA040 I/O error while requesting Ambari org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:176) org.apache.ambari.view.capacityscheduler.ConfigurationService.writeConfiguration(ConfigurationService.java:409)...

Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://xxx:8080/api/v1/clusters/AT/ sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627) java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:209)\n\t... 97 more\n"}

Explorer

Hi @Ian Roberts,

I know this is a bit of an old thread but I was wondering if you know if the syntax has changed at all with the new HDP version. I have been using HDP 2.5 sandbox and calling this gives me a 415 error indicating that application/json requests are not supported. Do you know how I can implement this programatically in the new HDP version?

Thanks,

John

Explorer

I have figured out a solution. For anyone who is interested in this particular problem the approach that I took was to run the following commands:

curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d @<json-file> http://<ambari-server>:8080/api/v1/clusters/<cluster-name>;

where <json-file> has the same structure as mentioned in @Ian Roberts solution. Note that 'tag' needs to have a unique value. The resource manager in yarn then needs to be restarted which can be done via:

curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d '{"ServiceComponentInfo":{"state":"INSTALLED"}}' http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/YARN/components/RESOURCEMANAGER

curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d '{"ServiceComponentInfo":{"state":"STARTED"}}' http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/YARN/components/RESOURCEMANAGER

All the best,

John

Expert Contributor
@Tim Veil

Please refer link for working command to achieve queue addition using script / in automated way -

https://community.hortonworks.com/questions/155903/how-to-add-new-yarn-queue-using-rest-api-ambari-c...