Support Questions
Find answers, ask questions, and share your expertise

How to add new yarn queue using Rest API / Ambari configs.sh script

Expert Contributor

Hi Team,

Is there any way to add new queue using Yarn rest api ? or Ambari configs.sh script ?

I tried below link but no luck -

https://community.hortonworks.com/questions/33578/api-to-manage-yarn-capacity-queue.html

http://<ambari_server>:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANC...


[root@node1 ~]#  curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d @cs.json http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic
HTTP/1.1 100 Continue


HTTP/1.1 400 Bad Request
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: no-store
Pragma: no-cache
Set-Cookie: AMBARISESSIONID=6a3wvwuvrhhu3e5258edqhjt;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
User: admin
Content-Type: text/plain
Content-Length: 217


{
  "status" : 400,
  "message" : "org.apache.ambari.server.controller.spi.UnsupportedPropertyException: The properties [items] specified in the request or predicate are not supported for the resource type Cluster."

When i tried to enable Developer Tools for Ambari View->Yarn queue manager , I see below call -

http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic/configurations?type=capacity-scheduler&tag=ver...

=======

{
  "href" : "http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic/configurations?type=capacity-scheduler&tag=version1514209101120",
  "items" : [
    {
      "href" : "http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic/configurations?type=capacity-scheduler&tag=version1514209101120",
      "tag" : "version1514209101120",
      "type" : "capacity-scheduler",
      "version" : 5,
      "Config" : {
        "cluster_name" : "hdp_mosaic",
        "stack_id" : "HDP-2.5"
      },
      "properties" : {
        "yarn.scheduler.capacity.maximum-am-resource-percent" : "0.2",
        "yarn.scheduler.capacity.maximum-applications" : "10000",
        "yarn.scheduler.capacity.node-locality-delay" : "40",
        "yarn.scheduler.capacity.queue-mappings-override.enable" : "false",
        "yarn.scheduler.capacity.resource-calculator" : "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
        "yarn.scheduler.capacity.root.MosaiqQueue.acl_administer_queue" : "*",
        "yarn.scheduler.capacity.root.MosaiqQueue.acl_submit_applications" : "*",
        "yarn.scheduler.capacity.root.MosaiqQueue.capacity" : "90",
        "yarn.scheduler.capacity.root.MosaiqQueue.maximum-capacity" : "90",
        "yarn.scheduler.capacity.root.MosaiqQueue.minimum-user-limit-percent" : "100",
        "yarn.scheduler.capacity.root.MosaiqQueue.ordering-policy" : "fifo",
        "yarn.scheduler.capacity.root.MosaiqQueue.state" : "RUNNING",
        "yarn.scheduler.capacity.root.MosaiqQueue.user-limit-factor" : "1",
        "yarn.scheduler.capacity.root.accessible-node-labels" : "*",
        "yarn.scheduler.capacity.root.acl_administer_queue" : "*",
        "yarn.scheduler.capacity.root.capacity" : "100",
        "yarn.scheduler.capacity.root.default.acl_submit_applications" : "*",
        "yarn.scheduler.capacity.root.default.capacity" : "10",
        "yarn.scheduler.capacity.root.default.maximum-capacity" : "100",
        "yarn.scheduler.capacity.root.default.state" : "RUNNING",
        "yarn.scheduler.capacity.root.default.user-limit-factor" : "1",
        "yarn.scheduler.capacity.root.queues" : "MosaiqQueue,default"
      }
    }
  ]
}
1 ACCEPTED SOLUTION

Expert Contributor
@pjoseph

I was able to achieve this using ambari api updating service configs.

Below is the working command - I have added Queue name - "MaxiqQueue"

 curl -u $ambari_user:$ambari_password -H 'X-Requested-By:admin' -X PUT "http://$ambari_server_host:8080/api/v1/clusters/$CLUSTER_NAME" -d '{
  "Clusters": {
    "desired_config": {
      "type": "capacity-scheduler",
      "tag": "version'$date'",
      "properties": {
            "yarn.scheduler.capacity.maximum-am-resource-percent" : "0.2",
            "yarn.scheduler.capacity.maximum-applications" : "10000",
            "yarn.scheduler.capacity.node-locality-delay" : "40",
            "yarn.scheduler.capacity.queue-mappings-override.enable" : "false",
            "yarn.scheduler.capacity.resource-calculator" : "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
            "yarn.scheduler.capacity.root.MaxiqQueue.acl_administer_queue" : "*",
            "yarn.scheduler.capacity.root.MaxiqQueue.acl_submit_applications" : "*",
            "yarn.scheduler.capacity.root.MaxiqQueue.capacity" : "90",
            "yarn.scheduler.capacity.root.MaxiqQueue.maximum-capacity" : "90",
            "yarn.scheduler.capacity.root.MaxiqQueue.minimum-user-limit-percent" : "100",
            "yarn.scheduler.capacity.root.MaxiqQueue.ordering-policy" : "fifo",
            "yarn.scheduler.capacity.root.MaxiqQueue.state" : "RUNNING",
            "yarn.scheduler.capacity.root.MaxiqQueue.user-limit-factor" : "1",
            "yarn.scheduler.capacity.root.accessible-node-labels" : "*",
            "yarn.scheduler.capacity.root.acl_administer_queue" : "yarn",
            "yarn.scheduler.capacity.root.capacity" : "100",
            "yarn.scheduler.capacity.root.default.acl_administer_queue" : "yarn",
            "yarn.scheduler.capacity.root.default.acl_submit_applications" : "yarn",
            "yarn.scheduler.capacity.root.default.capacity" : "10",
            "yarn.scheduler.capacity.root.default.maximum-capacity" : "100",
            "yarn.scheduler.capacity.root.default.state" : "RUNNING",
            "yarn.scheduler.capacity.root.default.user-limit-factor" : "1",
            "yarn.scheduler.capacity.root.queues" : "MaxiqQueue,default"
      }
    }
  }
}'

View solution in original post

10 REPLIES 10

Cloudera Employee

Hi @Sagar Shimpi, YARN does not have REST api to add a queue. We need to change the capacity-scheduler.xml and refresh yarn queues using "yarn rmadmin -refreshQueues" command. The same is done by the Ambari Server. The REST api you have shared is from Browser to Ambari Server, which internally calls the queue refresh command.

Expert Contributor
@pjoseph

I was able to achieve this using ambari api updating service configs.

Below is the working command - I have added Queue name - "MaxiqQueue"

 curl -u $ambari_user:$ambari_password -H 'X-Requested-By:admin' -X PUT "http://$ambari_server_host:8080/api/v1/clusters/$CLUSTER_NAME" -d '{
  "Clusters": {
    "desired_config": {
      "type": "capacity-scheduler",
      "tag": "version'$date'",
      "properties": {
            "yarn.scheduler.capacity.maximum-am-resource-percent" : "0.2",
            "yarn.scheduler.capacity.maximum-applications" : "10000",
            "yarn.scheduler.capacity.node-locality-delay" : "40",
            "yarn.scheduler.capacity.queue-mappings-override.enable" : "false",
            "yarn.scheduler.capacity.resource-calculator" : "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
            "yarn.scheduler.capacity.root.MaxiqQueue.acl_administer_queue" : "*",
            "yarn.scheduler.capacity.root.MaxiqQueue.acl_submit_applications" : "*",
            "yarn.scheduler.capacity.root.MaxiqQueue.capacity" : "90",
            "yarn.scheduler.capacity.root.MaxiqQueue.maximum-capacity" : "90",
            "yarn.scheduler.capacity.root.MaxiqQueue.minimum-user-limit-percent" : "100",
            "yarn.scheduler.capacity.root.MaxiqQueue.ordering-policy" : "fifo",
            "yarn.scheduler.capacity.root.MaxiqQueue.state" : "RUNNING",
            "yarn.scheduler.capacity.root.MaxiqQueue.user-limit-factor" : "1",
            "yarn.scheduler.capacity.root.accessible-node-labels" : "*",
            "yarn.scheduler.capacity.root.acl_administer_queue" : "yarn",
            "yarn.scheduler.capacity.root.capacity" : "100",
            "yarn.scheduler.capacity.root.default.acl_administer_queue" : "yarn",
            "yarn.scheduler.capacity.root.default.acl_submit_applications" : "yarn",
            "yarn.scheduler.capacity.root.default.capacity" : "10",
            "yarn.scheduler.capacity.root.default.maximum-capacity" : "100",
            "yarn.scheduler.capacity.root.default.state" : "RUNNING",
            "yarn.scheduler.capacity.root.default.user-limit-factor" : "1",
            "yarn.scheduler.capacity.root.queues" : "MaxiqQueue,default"
      }
    }
  }
}'

Explorer

@Sagar Shimpi

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT $ambari_server/api/v1/clusters/engsndbx -d '[{"Clusters":{ > "desired_config":[{ > "type":"capacity-scheduler", > "tag":"version1519845495539", > "properties":{ > "yarn.scheduler.capacity.maximum-am-resource-percent":"0.4", > "yarn.scheduler.capacity.maximum-applications":"10000", > "yarn.scheduler.capacity.node-locality-delay":"40", > "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator", > "yarn.scheduler.capacity.queue-mappings-override.enable":"false", > "yarn.scheduler.capacity.root.acl_administer_queue": "*", > "yarn.scheduler.capacity.root.capacity":"100", > "yarn.scheduler.capacity.root.queues": "Hive", > "yarn.scheduler.capacity.root.accessible-node-labels": "*", > "yarn.scheduler.capacity.root.Hive.acl_submit_applications": "*", > "yarn.scheduler.capacity.root.Hive.maximum-capacity":"100", > "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4", > "yarn.scheduler.capacity.root.Hive.state": "RUNNING", > "yarn.scheduler.capacity.root.Hive.capacity":"100"}, > "service_config_version_note":"New config version"}]}}]'

from above rest api I would like to change the "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4", but ending upwith error :

{ "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Configuration with tag 'version1519845495539' exists for 'capacity-scheduler'"

any suggestions ?

Explorer

Sorry I would like to update this "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4

Expert Contributor
@Veerendra Nath Jasthi

From the above error it seems the issue is with tag "versionxxxxx".

tag 'version1519845495539' exists for 'capacity-scheduler'"

Just update "tag" "version1519845495539" in curl command to some random number eg. version151984546666

Please retry and let me know if still any issue.

Explorer

Thank you so much it worked.....

Hi

Is it working for you? I tried the same thing, I am able to see the new queue configuration in Ambari queue manager UI, but need to manually click on "Save and refresh".

I see following warning in ambari server log "AbstractResourceProvider:506 - Can not determine request operation level. Operation level property should be specified for this request.

Explorer

@Sudhakara Vasiraju its working for me with out refreshing manually by calling below rest api....

curl -u admin:admin -H 'Content-Type:application/json' -H 'X-Requested-By:ambari' -iX PUT -d '{"save": "true"}' _http://localhost:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANCE/resources/scheduler/configuration/saveAndRefresh

Thank you @Veerendra Nath Jasthi ..

While adding queues I missed the property "yarn.scheduler.capacity.root.accessible-node-labels.<newnodelabelname>.capacity" property which was causing the issue.

@Veerendra Nath Jasthi As I am adding queues programmatically, I am hesitant to call API on views ... I am calling refresh queues api as follows, for which I need to manually do the save and refresh.

curl -u uname:pwd -H "X-Requested-By:ambari" -iX POST -d '{
  "RequestInfo" : {
    "command" : "REFRESHQUEUES",
    "context" : "Refresh YARN Capacity Scheduler",
    "operation_level" : {
        "level": "HOST_COMPONENT",
        "cluster_name": "<clustername>"
    },
    "parameters/forceRefreshConfigTags":"capacity-scheduler"
  },
  "Requests/resource_filters": [{
    "service_name" : "YARN",
    "component_name" : "RESOURCEMANAGER",
    "hosts" : "<hostname>"
  }]
}' http://<hostip>:<port>/api/v1/clusters/<clustername>/requests
 
; ;