How to add new yarn queue using Rest API / Ambari configs.sh script

[#1]

Hi Team,

Is there any way to add new queue using Yarn rest api ? or Ambari configs.sh script ?

I tried below link but no luck -

https://community.hortonworks.com/questions/33578/api-to-manage-yarn-capacity-queue.html

http://<ambari_server>:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANC...


[root@node1 ~]#  curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d @cs.json http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic
HTTP/1.1 100 Continue


HTTP/1.1 400 Bad Request
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: no-store
Pragma: no-cache
Set-Cookie: AMBARISESSIONID=6a3wvwuvrhhu3e5258edqhjt;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
User: admin
Content-Type: text/plain
Content-Length: 217


{
  "status" : 400,
  "message" : "org.apache.ambari.server.controller.spi.UnsupportedPropertyException: The properties [items] specified in the request or predicate are not supported for the resource type Cluster."

When i tried to enable Developer Tools for Ambari View->Yarn queue manager , I see below call -

http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic/configurations?type=capacity-scheduler&tag=ver...

=======

{
  "href" : "http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic/configurations?type=capacity-scheduler&tag=version1514209101120",
  "items" : [
    {
      "href" : "http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic/configurations?type=capacity-scheduler&tag=version1514209101120",
      "tag" : "version1514209101120",
      "type" : "capacity-scheduler",
      "version" : 5,
      "Config" : {
        "cluster_name" : "hdp_mosaic",
        "stack_id" : "HDP-2.5"
      },
      "properties" : {
        "yarn.scheduler.capacity.maximum-am-resource-percent" : "0.2",
        "yarn.scheduler.capacity.maximum-applications" : "10000",
        "yarn.scheduler.capacity.node-locality-delay" : "40",
        "yarn.scheduler.capacity.queue-mappings-override.enable" : "false",
        "yarn.scheduler.capacity.resource-calculator" : "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
        "yarn.scheduler.capacity.root.MosaiqQueue.acl_administer_queue" : "*",
        "yarn.scheduler.capacity.root.MosaiqQueue.acl_submit_applications" : "*",
        "yarn.scheduler.capacity.root.MosaiqQueue.capacity" : "90",
        "yarn.scheduler.capacity.root.MosaiqQueue.maximum-capacity" : "90",
        "yarn.scheduler.capacity.root.MosaiqQueue.minimum-user-limit-percent" : "100",
        "yarn.scheduler.capacity.root.MosaiqQueue.ordering-policy" : "fifo",
        "yarn.scheduler.capacity.root.MosaiqQueue.state" : "RUNNING",
        "yarn.scheduler.capacity.root.MosaiqQueue.user-limit-factor" : "1",
        "yarn.scheduler.capacity.root.accessible-node-labels" : "*",
        "yarn.scheduler.capacity.root.acl_administer_queue" : "*",
        "yarn.scheduler.capacity.root.capacity" : "100",
        "yarn.scheduler.capacity.root.default.acl_submit_applications" : "*",
        "yarn.scheduler.capacity.root.default.capacity" : "10",
        "yarn.scheduler.capacity.root.default.maximum-capacity" : "100",
        "yarn.scheduler.capacity.root.default.state" : "RUNNING",
        "yarn.scheduler.capacity.root.default.user-limit-factor" : "1",
        "yarn.scheduler.capacity.root.queues" : "MosaiqQueue,default"
      }
    }
  ]
}

[#2]

Hi @Sagar Shimpi, YARN does not have REST api to add a queue. We need to change the capacity-scheduler.xml and refresh yarn queues using "yarn rmadmin -refreshQueues" command. The same is done by the Ambari Server. The REST api you have shared is from Browser to Ambari Server, which internally calls the queue refresh command.

[#3 · Accepted solution]

@pjoseph

I was able to achieve this using ambari api updating service configs.

Below is the working command - I have added Queue name - "MaxiqQueue"

 curl -u $ambari_user:$ambari_password -H 'X-Requested-By:admin' -X PUT "http://$ambari_server_host:8080/api/v1/clusters/$CLUSTER_NAME" -d '{
  "Clusters": {
    "desired_config": {
      "type": "capacity-scheduler",
      "tag": "version'$date'",
      "properties": {
            "yarn.scheduler.capacity.maximum-am-resource-percent" : "0.2",
            "yarn.scheduler.capacity.maximum-applications" : "10000",
            "yarn.scheduler.capacity.node-locality-delay" : "40",
            "yarn.scheduler.capacity.queue-mappings-override.enable" : "false",
            "yarn.scheduler.capacity.resource-calculator" : "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
            "yarn.scheduler.capacity.root.MaxiqQueue.acl_administer_queue" : "*",
            "yarn.scheduler.capacity.root.MaxiqQueue.acl_submit_applications" : "*",
            "yarn.scheduler.capacity.root.MaxiqQueue.capacity" : "90",
            "yarn.scheduler.capacity.root.MaxiqQueue.maximum-capacity" : "90",
            "yarn.scheduler.capacity.root.MaxiqQueue.minimum-user-limit-percent" : "100",
            "yarn.scheduler.capacity.root.MaxiqQueue.ordering-policy" : "fifo",
            "yarn.scheduler.capacity.root.MaxiqQueue.state" : "RUNNING",
            "yarn.scheduler.capacity.root.MaxiqQueue.user-limit-factor" : "1",
            "yarn.scheduler.capacity.root.accessible-node-labels" : "*",
            "yarn.scheduler.capacity.root.acl_administer_queue" : "yarn",
            "yarn.scheduler.capacity.root.capacity" : "100",
            "yarn.scheduler.capacity.root.default.acl_administer_queue" : "yarn",
            "yarn.scheduler.capacity.root.default.acl_submit_applications" : "yarn",
            "yarn.scheduler.capacity.root.default.capacity" : "10",
            "yarn.scheduler.capacity.root.default.maximum-capacity" : "100",
            "yarn.scheduler.capacity.root.default.state" : "RUNNING",
            "yarn.scheduler.capacity.root.default.user-limit-factor" : "1",
            "yarn.scheduler.capacity.root.queues" : "MaxiqQueue,default"
      }
    }
  }
}'

[#4]

@Sagar Shimpi

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT $ambari_server/api/v1/clusters/engsndbx -d '[{"Clusters":{ > "desired_config":[{ > "type":"capacity-scheduler", > "tag":"version1519845495539", > "properties":{ > "yarn.scheduler.capacity.maximum-am-resource-percent":"0.4", > "yarn.scheduler.capacity.maximum-applications":"10000", > "yarn.scheduler.capacity.node-locality-delay":"40", > "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator", > "yarn.scheduler.capacity.queue-mappings-override.enable":"false", > "yarn.scheduler.capacity.root.acl_administer_queue": "*", > "yarn.scheduler.capacity.root.capacity":"100", > "yarn.scheduler.capacity.root.queues": "Hive", > "yarn.scheduler.capacity.root.accessible-node-labels": "*", > "yarn.scheduler.capacity.root.Hive.acl_submit_applications": "*", > "yarn.scheduler.capacity.root.Hive.maximum-capacity":"100", > "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4", > "yarn.scheduler.capacity.root.Hive.state": "RUNNING", > "yarn.scheduler.capacity.root.Hive.capacity":"100"}, > "service_config_version_note":"New config version"}]}}]'

from above rest api I would like to change the "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4", but ending upwith error :

{ "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Configuration with tag 'version1519845495539' exists for 'capacity-scheduler'"

any suggestions ?

[#5]

Sorry I would like to update this "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4

[#6]

@Veerendra Nath Jasthi

From the above error it seems the issue is with tag "versionxxxxx".

tag 'version1519845495539' exists for 'capacity-scheduler'"

Just update "tag" "version1519845495539" in curl command to some random number eg. version151984546666

Please retry and let me know if still any issue.

[#7]

Thank you so much it worked.....

[#8]

Hi

Is it working for you? I tried the same thing, I am able to see the new queue configuration in Ambari queue manager UI, but need to manually click on "Save and refresh".

I see following warning in ambari server log "AbstractResourceProvider:506 - Can not determine request operation level. Operation level property should be specified for this request.

[#9]

@Sudhakara Vasiraju its working for me with out refreshing manually by calling below rest api....

curl -u admin:admin -H 'Content-Type:application/json' -H 'X-Requested-By:ambari' -iX PUT -d '{"save": "true"}' _http://localhost:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANCE/resources/scheduler/configuration/saveAndRefresh

[#10]

Thank you @Veerendra Nath Jasthi ..

While adding queues I missed the property "yarn.scheduler.capacity.root.accessible-node-labels.<newnodelabelname>.capacity" property which was causing the issue.