Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari says Resource Manager is running, log files say otherwise

Highlighted

Ambari says Resource Manager is running, log files say otherwise

Explorer

My cluster seems to be having a problem with the App Timeline Server (ATS) and Resource Manager running at the same time.

As a test, I stopped both services, cleared out the log files, and just started RM. Ambari shows RM has started fine, but...

Here's what I found in the logs.

yarn-yarn-resourcemanager-<server>.log

2018-05-31 15:57:42,336 INFO  rmapp.RMAppImpl (RMAppImpl.java:handle(779)) - application_1527213246626_3635 State change from NEW to FAILED
2018-05-31 15:57:42,336 WARN  rmapp.RMAppImpl (RMAppImpl.java:<init>(423)) - The specific max attempts: 0 for application: 3636 is invalid, because it is out of the range [1, 2]. Use the global max attempts instead.
2018-05-31 15:57:42,336 INFO  rmapp.RMAppImpl (RMAppImpl.java:recover(792)) - Recovering app: application_1527213246626_3636 with 0 attempts and final state = FAILED

rm-audit.log

2018-05-31 15:57:49,435 WARN resourcemanager.RMAuditLogger: USER=dr.who OPERATION=Application Finished - Failed TARGET=RMAppManager     RESULT=FAILURE DESCRIPTION=App failed with state: FAILED        PERMISSIONS=org.apache.hadoop.security.AccessControlException: Queue root.default already has 1 applications, cannot accept submission of application: application_1527213246626_3508   APPID=application_1527213246626_3508

What's going on here?

8 REPLIES 8

Re: Ambari says Resource Manager is running, log files say otherwise

@Mike Wong ambari is correct when saying RM is running. RM will print information about applications running, being submitted or failing on the cluster among other things. It seems you need to check your yarn scheduler configuration as it seems your default queue will allow only 1 application to run at the time.

Queue root.default already has 1 applications, cannot accept submission of application

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: Ambari says Resource Manager is running, log files say otherwise

Explorer

@Felix Albani

Thanks for the input. Where do I check the yarn scheduler config?

Re: Ambari says Resource Manager is running, log files say otherwise

@Mike Wong I suggest you read over:

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-views/content/ch_using_yarn_queue...

and I also recommend you check the yarn.scheduler.minimum-allocation-mb (in case is too big), and yarn.scheduler.maximum-allocation-mb (in case is too small) - A good read here

https://community.hortonworks.com/questions/29469/yarn-container-size-flexible-to-satisfy-what-appli...

HTH

Re: Ambari says Resource Manager is running, log files say otherwise

@Mike Wong Yes, the default queue has no capacity. All capacity is assigned to queue llap. Usually when you submit an application without specifying the queue it will go to default queue. Since it does not have any capacity you run into the above problem. I suggest you give the default queue some capacity and take some from llap. Set default to 30 or so and llap to 70 - To move forward.

HTH

Re: Ambari says Resource Manager is running, log files say otherwise

Mentor

@Mike Wong

Can you share your config

Ambari UI-->Yarn-->Configs-->Advanced-->Capacity Scheduler

or see attached screenshot


rm.png

Re: Ambari says Resource Manager is running, log files say otherwise

Explorer

@Geoffrey Shelton Okot @Felix Albani

Capacity Scheduler config

77471-screen-shot-2018-05-31-at-33432-pm.png

Yarn Queue Manager-

77472-screen-shot-2018-05-31-at-33259-pm.png

Are there any obviously wrong settings?

Re: Ambari says Resource Manager is running, log files say otherwise

Mentor

@Mike Wong

Just to be sure did you enable YARN pre-emption?

Re: Ambari says Resource Manager is running, log files say otherwise

Explorer

@Geoffrey Shelton Okot

Right now, yarn.resourcemanager.scheduler.monitor.enable is set to false.

Looks like there are few settings I'll have to add to yarn-site.xml:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_yarn-resource-management/content/preempt...

I'll get to it...

Don't have an account?
Coming from Hortonworks? Activate your account here