question Re: Mapreduce job hang, waiting for AM container to be allocated. in Archives of Support Questions (Read Only)

Mapreduce job hang, waiting for AM container to be allocated.

Nilesh — Sun, 18 Aug 2019 11:04:28 GMT

Hi Team,

Job hang while importing tables via sqoop shown following message in web UI.

ACCEPTED: waiting for AM container to be allocated, launched and register with RM

Kindly suggest.

Re: Mapreduce job hang, waiting for AM container to be allocated.

vxu — Wed, 13 Apr 2016 20:04:51 GMT

Hi, Nilesh. Have you checked the yarn scheduler? Is the default queue out of resource?

Re: Mapreduce job hang, waiting for AM container to be allocated.

sshimpi — Wed, 13 Apr 2016 20:10:20 GMT

@Nilesh

I will suggest yo revisit your yarn memory configuration once. Seems you might be running out of memory.

Can you please let me know what values you have set for yarn.scheduler.capacity.maximum-am-resource-percent ?

If possible please attach -

1. Yarn RM UI snap

2. yarn-site.xml

3. mapred-site.xml

4. Scheduler snap from RM UI -

http://<RM-Hostname>:8088/cluster/scheduler

Re: Mapreduce job hang, waiting for AM container to be allocated.

Nilesh — Sun, 18 Aug 2019 11:04:20 GMT

@Sagar Shimpi

yarn.scheduler.capacity.maximum-am-resource-percent=0.2

Kindly find attached file for reference.

yarn-site.xml

mapred-site.xml

Re: Mapreduce job hang, waiting for AM container to be allocated.

sshimpi — Wed, 13 Apr 2016 21:56:46 GMT

@Nilesh From the resource manager UI i see there are no "Active Nodes" running and hence the "Total Memory" in UI is displaying 0.

Can you check if your node managers are UP and communicating to RM.

Re: Mapreduce job hang, waiting for AM container to be allocated.

KuldeepK — Thu, 14 Apr 2016 01:23:42 GMT

@Nilesh

There are 2 unhealthy nodes, if you click on "2" under unhealthy nodes section, you will get reason why they are unhealthy, it could be because of bad disk etc. Please check and try to check nodemanager's logs, you will get more info in there.

Re: Mapreduce job hang, waiting for AM container to be allocated.

Nilesh — Thu, 14 Apr 2016 13:33:55 GMT

@Kuldeep Kulkarni and @Sagar Shimpi

Issue has been resolved by changing below parameter in yarn-site.xml

yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage

Previously it was 90% I changed it to 99 Now Job is in running state.

Could you please shed some light on this parameter.

Re: Mapreduce job hang, waiting for AM container to be allocated.

KuldeepK — Thu, 14 Apr 2016 13:48:26 GMT

@Nilesh -

yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage

The maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0. If the value is greater than or equal to 100, the nodemanager will check for full disk. This applies to yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs.

Re: Mapreduce job hang, waiting for AM container to be allocated.

Nilesh — Thu, 14 Apr 2016 14:01:15 GMT

@Kuldeep Kulkarni

How to find which disk is marked as bad?

Re: Mapreduce job hang, waiting for AM container to be allocated.

KuldeepK — Sun, 18 Aug 2019 11:04:07 GMT

Generally it should be visible on RM UI once you click on the unhealthy nodes.

You can go the unhealthy node(http://<unhealthy-node-manager>:8042/jmx) and check JMX

Re: Mapreduce job hang, waiting for AM container to be allocated.

cherukuri_49 — Wed, 11 Jan 2017 21:26:23 GMT

Hi @Nilesh,

I am too facing same issue, could you please suggest me if you have solution.

Re: Mapreduce job hang, waiting for AM container to be allocated.

smkmuthu — Fri, 03 Aug 2018 21:52:46 GMT

Hi All

I am also facing the same issue. but My server hard disk is new. There is no warning/alert in my Ambari level.

Once my current job completes, then only the second job is allowed to execute..

If my first job is running for 60 minutes, my second job is on hold. Any suggestions.

server capacity : 16 core , 64 GB RAM

Thx

Muthu

Re: Mapreduce job hang, waiting for AM container to be allocated.

volker_kuechenh — Tue, 04 Sep 2018 22:22:49 GMT

Hi, this Looks like fifo-scheduling / capacity scheduling with 1 group only Try to switch to fair scheduling in yarn.

Regards,

Volker