question How to add new yarn queue using Rest API / Ambari configs.sh script in Archives of Support Questions (Read Only)

How to add new yarn queue using Rest API / Ambari configs.sh script

sagarshimpi — Mon, 25 Dec 2017 21:31:34 GMT

Hi Team,

Is there any way to add new queue using Yarn rest api ? or Ambari configs.sh script ?

I tried below link but no luck -

https://community.hortonworks.com/questions/33578/api-to-manage-yarn-capacity-queue.html

http://<ambari_server>:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANCE/resources/scheduler/configuration


[root@node1 ~]#  curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d @cs.json http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic
HTTP/1.1 100 Continue


HTTP/1.1 400 Bad Request
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: no-store
Pragma: no-cache
Set-Cookie: AMBARISESSIONID=6a3wvwuvrhhu3e5258edqhjt;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
User: admin
Content-Type: text/plain
Content-Length: 217


{
  "status" : 400,
  "message" : "org.apache.ambari.server.controller.spi.UnsupportedPropertyException: The properties [items] specified in the request or predicate are not supported for the resource type Cluster."

When i tried to enable Developer Tools for Ambari View->Yarn queue manager , I see below call -

http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic/configurations?type=capacity-scheduler&tag=version1514209101120

=======

{
  "href" : "http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic/configurations?type=capacity-scheduler&tag=version1514209101120",
  "items" : [
    {
      "href" : "http://192.168.56.111:8080/api/v1/clusters/hdp_mosaic/configurations?type=capacity-scheduler&tag=version1514209101120",
      "tag" : "version1514209101120",
      "type" : "capacity-scheduler",
      "version" : 5,
      "Config" : {
        "cluster_name" : "hdp_mosaic",
        "stack_id" : "HDP-2.5"
      },
      "properties" : {
        "yarn.scheduler.capacity.maximum-am-resource-percent" : "0.2",
        "yarn.scheduler.capacity.maximum-applications" : "10000",
        "yarn.scheduler.capacity.node-locality-delay" : "40",
        "yarn.scheduler.capacity.queue-mappings-override.enable" : "false",
        "yarn.scheduler.capacity.resource-calculator" : "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
        "yarn.scheduler.capacity.root.MosaiqQueue.acl_administer_queue" : "*",
        "yarn.scheduler.capacity.root.MosaiqQueue.acl_submit_applications" : "*",
        "yarn.scheduler.capacity.root.MosaiqQueue.capacity" : "90",
        "yarn.scheduler.capacity.root.MosaiqQueue.maximum-capacity" : "90",
        "yarn.scheduler.capacity.root.MosaiqQueue.minimum-user-limit-percent" : "100",
        "yarn.scheduler.capacity.root.MosaiqQueue.ordering-policy" : "fifo",
        "yarn.scheduler.capacity.root.MosaiqQueue.state" : "RUNNING",
        "yarn.scheduler.capacity.root.MosaiqQueue.user-limit-factor" : "1",
        "yarn.scheduler.capacity.root.accessible-node-labels" : "*",
        "yarn.scheduler.capacity.root.acl_administer_queue" : "*",
        "yarn.scheduler.capacity.root.capacity" : "100",
        "yarn.scheduler.capacity.root.default.acl_submit_applications" : "*",
        "yarn.scheduler.capacity.root.default.capacity" : "10",
        "yarn.scheduler.capacity.root.default.maximum-capacity" : "100",
        "yarn.scheduler.capacity.root.default.state" : "RUNNING",
        "yarn.scheduler.capacity.root.default.user-limit-factor" : "1",
        "yarn.scheduler.capacity.root.queues" : "MosaiqQueue,default"
      }
    }
  ]
}

Re: How to add new yarn queue using Rest API / Ambari configs.sh script

pjoseph — Tue, 09 Jan 2018 19:10:42 GMT

Hi @Sagar Shimpi, YARN does not have REST api to add a queue. We need to change the capacity-scheduler.xml and refresh yarn queues using "yarn rmadmin -refreshQueues" command. The same is done by the Ambari Server. The REST api you have shared is from Browser to Ambari Server, which internally calls the queue refresh command.

Re: How to add new yarn queue using Rest API / Ambari configs.sh script

sagarshimpi — Thu, 01 Mar 2018 14:22:19 GMT

@pjoseph

I was able to achieve this using ambari api updating service configs.

Below is the working command - I have added Queue name - "MaxiqQueue"

 curl -u $ambari_user:$ambari_password -H 'X-Requested-By:admin' -X PUT "http://$ambari_server_host:8080/api/v1/clusters/$CLUSTER_NAME" -d '{
  "Clusters": {
    "desired_config": {
      "type": "capacity-scheduler",
      "tag": "version'$date'",
      "properties": {
            "yarn.scheduler.capacity.maximum-am-resource-percent" : "0.2",
            "yarn.scheduler.capacity.maximum-applications" : "10000",
            "yarn.scheduler.capacity.node-locality-delay" : "40",
            "yarn.scheduler.capacity.queue-mappings-override.enable" : "false",
            "yarn.scheduler.capacity.resource-calculator" : "org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator",
            "yarn.scheduler.capacity.root.MaxiqQueue.acl_administer_queue" : "*",
            "yarn.scheduler.capacity.root.MaxiqQueue.acl_submit_applications" : "*",
            "yarn.scheduler.capacity.root.MaxiqQueue.capacity" : "90",
            "yarn.scheduler.capacity.root.MaxiqQueue.maximum-capacity" : "90",
            "yarn.scheduler.capacity.root.MaxiqQueue.minimum-user-limit-percent" : "100",
            "yarn.scheduler.capacity.root.MaxiqQueue.ordering-policy" : "fifo",
            "yarn.scheduler.capacity.root.MaxiqQueue.state" : "RUNNING",
            "yarn.scheduler.capacity.root.MaxiqQueue.user-limit-factor" : "1",
            "yarn.scheduler.capacity.root.accessible-node-labels" : "*",
            "yarn.scheduler.capacity.root.acl_administer_queue" : "yarn",
            "yarn.scheduler.capacity.root.capacity" : "100",
            "yarn.scheduler.capacity.root.default.acl_administer_queue" : "yarn",
            "yarn.scheduler.capacity.root.default.acl_submit_applications" : "yarn",
            "yarn.scheduler.capacity.root.default.capacity" : "10",
            "yarn.scheduler.capacity.root.default.maximum-capacity" : "100",
            "yarn.scheduler.capacity.root.default.state" : "RUNNING",
            "yarn.scheduler.capacity.root.default.user-limit-factor" : "1",
            "yarn.scheduler.capacity.root.queues" : "MaxiqQueue,default"
      }
    }
  }
}'

Re: How to add new yarn queue using Rest API / Ambari configs.sh script

Jasthi — Fri, 02 Mar 2018 01:14:18 GMT

@Sagar Shimpi

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT $ambari_server/api/v1/clusters/engsndbx -d '[{"Clusters":{ > "desired_config":[{ > "type":"capacity-scheduler", > "tag":"version1519845495539", > "properties":{ > "yarn.scheduler.capacity.maximum-am-resource-percent":"0.4", > "yarn.scheduler.capacity.maximum-applications":"10000", > "yarn.scheduler.capacity.node-locality-delay":"40", > "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator", > "yarn.scheduler.capacity.queue-mappings-override.enable":"false", > "yarn.scheduler.capacity.root.acl_administer_queue": "*", > "yarn.scheduler.capacity.root.capacity":"100", > "yarn.scheduler.capacity.root.queues": "Hive", > "yarn.scheduler.capacity.root.accessible-node-labels": "*", > "yarn.scheduler.capacity.root.Hive.acl_submit_applications": "*", > "yarn.scheduler.capacity.root.Hive.maximum-capacity":"100", > "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4", > "yarn.scheduler.capacity.root.Hive.state": "RUNNING", > "yarn.scheduler.capacity.root.Hive.capacity":"100"}, > "service_config_version_note":"New config version"}]}}]'

from above rest api I would like to change the "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4", but ending upwith error :

{ "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Configuration with tag 'version1519845495539' exists for 'capacity-scheduler'"

any suggestions ?

Re: How to add new yarn queue using Rest API / Ambari configs.sh script

Jasthi — Fri, 02 Mar 2018 01:18:15 GMT

Sorry I would like to update this "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4

Re: How to add new yarn queue using Rest API / Ambari configs.sh script

sagarshimpi — Fri, 02 Mar 2018 04:28:59 GMT

@Veerendra Nath Jasthi

From the above error it seems the issue is with tag "versionxxxxx".

tag 'version1519845495539' exists for 'capacity-scheduler'"

Just update "tag" "version1519845495539" in curl command to some random number eg. version151984546666

Please retry and let me know if still any issue.

Re: How to add new yarn queue using Rest API / Ambari configs.sh script

Jasthi — Fri, 02 Mar 2018 04:31:17 GMT

Thank you so much it worked.....

Re: How to add new yarn queue using Rest API / Ambari configs.sh script

sudhavasiraju — Sat, 17 Mar 2018 01:16:48 GMT

Is it working for you? I tried the same thing, I am able to see the new queue configuration in Ambari queue manager UI, but need to manually click on "Save and refresh".

I see following warning in ambari server log "AbstractResourceProvider:506 - Can not determine request operation level. Operation level property should be specified for this request.

Re: How to add new yarn queue using Rest API / Ambari configs.sh script

Jasthi — Sat, 17 Mar 2018 01:22:49 GMT

@Sudhakara Vasiraju its working for me with out refreshing manually by calling below rest api....

curl -u admin:admin -H 'Content-Type:application/json' -H 'X-Requested-By:ambari' -iX PUT -d '{"save": "true"}' _http://localhost:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANCE/resources/scheduler/configuration/saveAndRefresh

Re: How to add new yarn queue using Rest API / Ambari configs.sh script

sudhavasiraju — Sat, 17 Mar 2018 02:56:31 GMT

@Veerendra Nath Jasthi As I am adding queues programmatically, I am hesitant to call API on views ... I am calling refresh queues api as follows, for which I need to manually do the save and refresh.

curl -u uname:pwd -H "X-Requested-By:ambari" -iX POST -d '{
  "RequestInfo" : {
    "command" : "REFRESHQUEUES",
    "context" : "Refresh YARN Capacity Scheduler",
    "operation_level" : {
        "level": "HOST_COMPONENT",
        "cluster_name": "<clustername>"
    },
    "parameters/forceRefreshConfigTags":"capacity-scheduler"
  },
  "Requests/resource_filters": [{
    "service_name" : "YARN",
    "component_name" : "RESOURCEMANAGER",
    "hosts" : "<hostname>"
  }]
}' http://<hostip>:<port>/api/v1/clusters/<clustername>/requests

Re: How to add new yarn queue using Rest API / Ambari configs.sh script

sudhavasiraju — Tue, 20 Mar 2018 21:58:10 GMT

Thank you @Veerendra Nath Jasthi ..

While adding queues I missed the property "yarn.scheduler.capacity.root.accessible-node-labels.<newnodelabelname>.capacity" property which was causing the issue.