Support Questions
Find answers, ask questions, and share your expertise

Mapreduce job hang, waiting for AM container to be allocated.

Solved Go to solution
Highlighted

Mapreduce job hang, waiting for AM container to be allocated.

Contributor

Hi Team,

Job hang while importing tables via sqoop shown following message in web UI.

ACCEPTED: waiting for AM container to be allocated, launched and register with RM

Kindly suggest.

3413-sqoop-mapreduce.jpg

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Mapreduce job hang, waiting for AM container to be allocated.

Super Guru
@Nilesh

There are 2 unhealthy nodes, if you click on "2" under unhealthy nodes section, you will get reason why they are unhealthy, it could be because of bad disk etc. Please check and try to check nodemanager's logs, you will get more info in there.

View solution in original post

12 REPLIES 12

Re: Mapreduce job hang, waiting for AM container to be allocated.

Contributor

Hi, Nilesh. Have you checked the yarn scheduler? Is the default queue out of resource?

Highlighted

Re: Mapreduce job hang, waiting for AM container to be allocated.

@Nilesh

I will suggest yo revisit your yarn memory configuration once. Seems you might be running out of memory.

Can you please let me know what values you have set for yarn.scheduler.capacity.maximum-am-resource-percent ?

If possible please attach -

1. Yarn RM UI snap

2. yarn-site.xml

3. mapred-site.xml

4. Scheduler snap from RM UI -

http://<RM-Hostname>:8088/cluster/scheduler

Highlighted

Re: Mapreduce job hang, waiting for AM container to be allocated.

Contributor

@Sagar Shimpi

yarn.scheduler.capacity.maximum-am-resource-percent=0.2

Kindly find attached file for reference.

3415-scheduler-snap.jpg

3418-yarn-memory.jpg

yarn-site.xml

mapred-site.xml

Highlighted

Re: Mapreduce job hang, waiting for AM container to be allocated.

@Nilesh From the resource manager UI i see there are no "Active Nodes" running and hence the "Total Memory" in UI is displaying 0.

Can you check if your node managers are UP and communicating to RM.

Highlighted

Re: Mapreduce job hang, waiting for AM container to be allocated.

Super Guru
@Nilesh

There are 2 unhealthy nodes, if you click on "2" under unhealthy nodes section, you will get reason why they are unhealthy, it could be because of bad disk etc. Please check and try to check nodemanager's logs, you will get more info in there.

View solution in original post

Highlighted

Re: Mapreduce job hang, waiting for AM container to be allocated.

Contributor

@Kuldeep Kulkarni and @Sagar Shimpi

Issue has been resolved by changing below parameter in yarn-site.xml

yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage

Previously it was 90% I changed it to 99 Now Job is in running state.

Could you please shed some light on this parameter.

Highlighted

Re: Mapreduce job hang, waiting for AM container to be allocated.

Super Guru

@Nilesh -

yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage

The maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0. If the value is greater than or equal to 100, the nodemanager will check for full disk. This applies to yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs.

Highlighted

Re: Mapreduce job hang, waiting for AM container to be allocated.

Contributor

@Kuldeep Kulkarni

How to find which disk is marked as bad?

Highlighted

Re: Mapreduce job hang, waiting for AM container to be allocated.

Super Guru

Generally it should be visible on RM UI once you click on the unhealthy nodes.

OR

You can go the unhealthy node(http://<unhealthy-node-manager>:8042/jmx) and check JMX

3443-screen-shot-2016-04-14-at-125436-pm.png