Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Accepted status of hive jobs in resource manager.

Highlighted

Accepted status of hive jobs in resource manager.

New Contributor

HI,

I have one node manager and using just 5.4% of the queue; however, tez jobs always stay in accepted status. The following error is showing in the diagnostics tab. I don;t have any issues with Spark Jobs.

Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:50176, vCores:1>; Queue Resource Limit for AM = <memory:35840, vCores:1>; User AM Resource Limit of the queue = <memory:35840, vCores:1>; Queue AM Resource Usage = <memory:5120, vCores:1>;

I have increased the am resource on scheduler.capacity to 40% and even 60% but still no luck. The config of capacity scheduler is below, Any thoughts?

yarn.scheduler.capacity.maximum-am-resource-percent=0.6 yarn.scheduler.capacity.maximum-applications=30 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.queue-mappings-override.enable=false yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator yarn.scheduler.capacity.root.Training.acl_submit_applications=* yarn.scheduler.capacity.root.Training.capacity=60 yarn.scheduler.capacity.root.Training.maximum-am-resource-percent=0.4 yarn.scheduler.capacity.root.Training.maximum-capacity=100 yarn.scheduler.capacity.root.Training.minimum-user-limit-percent=10 yarn.scheduler.capacity.root.Training.ordering-policy=fifo yarn.scheduler.capacity.root.Training.priority=1 yarn.scheduler.capacity.root.Training.state=RUNNING yarn.scheduler.capacity.root.Training.user-limit-factor=1 yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.acl_submit_applications=* yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.default.acl_administer_queue=* yarn.scheduler.capacity.root.default.acl_submit_applications=* yarn.scheduler.capacity.root.default.capacity=20 yarn.scheduler.capacity.root.default.maximum-capacity=20 yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.default.ordering-policy=fifo yarn.scheduler.capacity.root.default.priority=0 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.user-limit-factor=1 yarn.scheduler.capacity.root.llap.acl_administer_queue=* yarn.scheduler.capacity.root.llap.acl_submit_applications=* yarn.scheduler.capacity.root.llap.capacity=20 yarn.scheduler.capacity.root.llap.maximum-capacity=20 yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.llap.priority=0 yarn.scheduler.capacity.root.llap.state=RUNNING yarn.scheduler.capacity.root.llap.user-limit-factor=1 yarn.scheduler.capacity.root.maximum-capacity=100 yarn.scheduler.capacity.root.ordering-policy=priority-utilization yarn.scheduler.capacity.root.priority=0 yarn.scheduler.capacity.root.queues=Training,default,llap

I would really appreciate your help.

M.J

4 REPLIES 4

Re: Accepted status of hive jobs in resource manager.

Cloudera Employee

Hi @Mahsa Jan,

If you go to the resourcemanager --> click on the <app ID> -> click on Logs --> click on syslog

what do you see, can you share the output?

Regards,

AQ

Re: Accepted status of hive jobs in resource manager.

New Contributor

Hi Aquilodran,

Please find the following error under diagnostic tab. These is no error under logs.

Diagnostics [Wed Dec 05 16:44:03 -0700 2018] Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:73728, vCores:1>; Queue Resource Limit for AM = <memory:98304, vCores:1>; User AM Resource Limit of the queue = <memory:49152, vCores:1>; Queue AM Resource Usage = <memory:49152, vCores:2>;
Outstanding Resource Requests
Scheduler KeyResource NameCapability# ContainersRelax LocalityNode Label Expression
0*<Memory:73728;vCores:>1trueN/A
Scheduling Info
Allocated ResourceRunning ContainersPreempted ResourceNum Non-AM container preemptedNum AM container preemptedAggregated Resource Usage
0 MBs, 0 VCores00 MBs, 0 VCores000 MBs, 0 VCores (× Secs)
,

Hi Aquilodran,

I see this under dignostices. No data under logs.

Diagnostics [Wed Dec 05 16:44:03 -0700 2018] Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:73728, vCores:1>; Queue Resource Limit for AM = <memory:98304, vCores:1>; User AM Resource Limit of the queue = <memory:49152, vCores:1>; Queue AM Resource Usage = <memory:49152, vCores:2>;
Outstanding Resource Requests
Scheduler KeyResource NameCapability# ContainersRelax LocalityNode Label Expression
0*<Memory:73728;vCores:>1trueN/A
Scheduling Info
Allocated ResourceRunning ContainersPreempted ResourceNum Non-AM container preemptedNum AM container preemptedAggregated Resource Usage
0 MBs, 0 VCores00 MBs, 0 VCores000 MBs, 0 VCores (× Secs)

Re: Accepted status of hive jobs in resource manager.

New Contributor

And this is my latest capacity scheduler config

yarn.scheduler.capacity.maximum-am-resource-percent=0.4 yarn.scheduler.capacity.maximum-applications=1000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.queue-mappings-override.enable=false yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator yarn.scheduler.capacity.root.Training.acl_submit_applications=* yarn.scheduler.capacity.root.Training.capacity=20 yarn.scheduler.capacity.root.Training.maximum-am-resource-percent=0.4 yarn.scheduler.capacity.root.Training.maximum-capacity=60 yarn.scheduler.capacity.root.Training.minimum-user-limit-percent=10 yarn.scheduler.capacity.root.Training.ordering-policy=fifo yarn.scheduler.capacity.root.Training.priority=1 yarn.scheduler.capacity.root.Training.state=RUNNING yarn.scheduler.capacity.root.Training.user-limit-factor=1 yarn.scheduler.capacity.root.accessible-node-labels=* yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.acl_submit_applications=* yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.default.acl_administer_queue=* yarn.scheduler.capacity.root.default.acl_submit_applications=* yarn.scheduler.capacity.root.default.capacity=20 yarn.scheduler.capacity.root.default.maximum-capacity=20 yarn.scheduler.capacity.root.default.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.default.ordering-policy=fifo yarn.scheduler.capacity.root.default.priority=0 yarn.scheduler.capacity.root.default.state=RUNNING yarn.scheduler.capacity.root.default.user-limit-factor=1 yarn.scheduler.capacity.root.llap.acl_administer_queue=* yarn.scheduler.capacity.root.llap.acl_submit_applications=* yarn.scheduler.capacity.root.llap.capacity=60 yarn.scheduler.capacity.root.llap.maximum-capacity=60 yarn.scheduler.capacity.root.llap.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.llap.priority=0 yarn.scheduler.capacity.root.llap.state=RUNNING yarn.scheduler.capacity.root.llap.user-limit-factor=1 yarn.scheduler.capacity.root.maximum-capacity=100 yarn.scheduler.capacity.root.ordering-policy=priority-utilization yarn.scheduler.capacity.root.priority=0 yarn.scheduler.capacity.root.queues=Training,default,llap

Re: Accepted status of hive jobs in resource manager.

Cloudera Employee

Hi @Mahsa Jan,

Could you please check in resource manager UI the status of "Active Nodes" and "Unhealthy Nodes"

and also check: yarn node -list

Regards,

AQ

Don't have an account?
Coming from Hortonworks? Activate your account here