Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

YARN ResourceManager REST API throwing erro?

Highlighted

YARN ResourceManager REST API throwing erro?

Explorer

I was trying change the configuration of Capacity-scheduler on the fly using rest-api call. Below is the configuration look like:

curl -v -u admin:admin -H "Content-Type: application/json" -H "X-Requested-By:ambari" -X PUT http://localhost:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/AUTO_CS_INSTANCE/reso... --data '{ "Clusters": { "desired_config": [ { "type": "capacity-scheduler", "tag": "version14534007568115", "service_config_version_note": "To test", "properties": { "yarn.scheduler.capacity.maximum-am-resource-percent": 0.2, "yarn.scheduler.capacity.maximum-applications": 10000, "yarn.scheduler.capacity.node-locality-delay": 40, "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator", "yarn.scheduler.capacity.queue-mappings-override.enable": false, "yarn.scheduler.capacity.root.acl_administer_queue": "*", "yarn.scheduler.capacity.root.capacity": 100, "yarn.scheduler.capacity.root.queues": "Hive", "yarn.scheduler.capacity.root.accessible-node-labels": "*", "yarn.scheduler.capacity.root.Hive.acl_submit_applications": "*", "yarn.scheduler.capacity.root.Hive.maximum-capacity": 100, "yarn.scheduler.capacity.root.Hive.user-limit-factor": 4, "yarn.scheduler.capacity.root.Hive.state": "RUNNING", "yarn.scheduler.capacity.root.Hive.capacity": 100 } } ] } }'

getting error like :

About to connect() to localhost port 8080 (#0) * Trying localhost... Connection refused * couldn't connect to host * Closing connection #0 curl: (7) couldn't connect to host

In our env. we dont have kerberos enabled.

5 REPLIES 5
Highlighted

Re: YARN ResourceManager REST API throwing erro?

Rising Star

@Veerendra Nath Jasthi, is your Ambari Server running on localhost?

Highlighted

Re: YARN ResourceManager REST API throwing erro?

Rising Star

Note, you are hitting an Ambari REST API and not RM REST API.

Highlighted

Re: YARN ResourceManager REST API throwing erro?

Explorer

@Gour Saha

https://community.hortonworks.com/content/supportkb/49421/how-to-administer-capacity-scheduler-view-...

the above link says that we should hit ambari to change the configurations on the fly.

No I do have ip address but cant share that so why I named it as a localhost whenever I am trying to call I will have ip in place of localhost that ambari server is running...

Highlighted

Re: YARN ResourceManager REST API throwing erro?

Explorer

Hi Saha I was able to run this but ending up error...

RESTAPI :

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT $ambari_server/api/v1/clusters/engsndbx -d '[{"Clusters":{ > "desired_config":[{ > "type":"capacity-scheduler", > "tag":"version1519845495539", > "properties":{ > "yarn.scheduler.capacity.maximum-am-resource-percent":"0.4", > "yarn.scheduler.capacity.maximum-applications":"10000", > "yarn.scheduler.capacity.node-locality-delay":"40", > "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator", > "yarn.scheduler.capacity.queue-mappings-override.enable":"false", > "yarn.scheduler.capacity.root.acl_administer_queue": "*", > "yarn.scheduler.capacity.root.capacity":"100", > "yarn.scheduler.capacity.root.queues": "Hive", > "yarn.scheduler.capacity.root.accessible-node-labels": "*", > "yarn.scheduler.capacity.root.Hive.acl_submit_applications": "*", > "yarn.scheduler.capacity.root.Hive.maximum-capacity":"100", > "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4", > "yarn.scheduler.capacity.root.Hive.state": "RUNNING", > "yarn.scheduler.capacity.root.Hive.capacity":"100"}, > "service_config_version_note":"New config version"}]}}]'

from above rest api I would like to update the "yarn.scheduler.capacity.root.Hive.user-limit-factor":"4", but ending upwith error :

{ "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Configuration with tag 'version1519845495539' exists for 'capacity-scheduler'"

any suggestions ?

Highlighted

Re: YARN ResourceManager REST API throwing erro?

Rising Star

You need to use a new tag, not one of the existing ones. Typically it is the "version" keyword followed by current timestamp.

If you don't absolutely need to use the REST API or you don't want to deal with the version tag, you should use configs.sh/configs.py (the sh is not supported in some older Ambari versions).

Sample get call is -

/var/lib/ambari-server/resources/scripts/configs.py -a get -l <ambari_server_host> -n <cluster_name> -c capacity-scheduler -f /tmp/cs.json

Sample output in /tmp/cs.json is -

{
  "properties": {
    "yarn.scheduler.capacity.maximum-am-resource-percent": "0.4", 
    "yarn.scheduler.capacity.maximum-applications": "10000", 
    "yarn.scheduler.capacity.node-locality-delay": "40", 
    "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator", 
    "yarn.scheduler.capacity.queue-mappings-override.enable": "false", 
    "yarn.scheduler.capacity.root.acl_administer_queue": "*", 
    "yarn.scheduler.capacity.root.capacity": "100", 
    "yarn.scheduler.capacity.root.queues": "Hive", 
    "yarn.scheduler.capacity.root.accessible-node-labels": "*", 
    "yarn.scheduler.capacity.root.Hive.acl_submit_applications": "*", 
    "yarn.scheduler.capacity.root.Hive.maximum-capacity": "100", 
    "yarn.scheduler.capacity.root.Hive.user-limit-factor": "4", 
    "yarn.scheduler.capacity.root.Hive.state": "RUNNING", 
    "yarn.scheduler.capacity.root.Hive.capacity": "100"
  }
}

Help is -

/var/lib/ambari-server/resources/scripts/configs.py -h

To perform the change you want to make, edit the file /tmp/cs.json with your desired change (update value of yarn.scheduler.capacity.root.Hive.user-limit-factor in your case) then use the "-a set" option with the same file. Sample cmd provided below -

/var/lib/ambari-server/resources/scripts/configs.py -a set -l <ambari_server_host> -n <cluster_name>-c capacity-scheduler -f /tmp/cs1.json

Note, you need to refresh queues to make this change take effect. You can do it by running rmadmin via command line -

yarn rmadmin -refreshQueues

Or, use the Ambari REST API -

curl -u admin:admin -H 'Content-Type:application/json' -H 'X-Requested-By:ambari' -iX PUT -d '{"save": "true"}' http://<ambari-server>:8080/api/v1/views/CAPACITY-SCHEDULER/versions/1.0.0/instances/<view_instance_...

You do not need to restart RM for capacity scheduler changes. However, if you make changes to other configs like yarn-site via configs.py you need to restart RM. You can do so by using Ambari REST APIs as shown below.

Stop RM -

curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d '{"ServiceComponentInfo":{"state":"INSTALLED"}}' http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/YARN/components/RESOURCEMANAGER

Start RM -

curl -u admin:admin -H "X-Requested-By:ambari" -iX PUT -d '{"ServiceComponentInfo":{"state":"STARTED"}}' http://<ambari-server>:8080/api/v1/clusters/<cluster-name>/services/YARN/components/RESOURCEMANAGER
Don't have an account?
Coming from Hortonworks? Activate your account here