Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Mapreduce job hang, waiting for AM container to be allocated.

avatar
Expert Contributor

Hi Team,

Job hang while importing tables via sqoop shown following message in web UI.

ACCEPTED: waiting for AM container to be allocated, launched and register with RM

Kindly suggest.

3413-sqoop-mapreduce.jpg

1 ACCEPTED SOLUTION

avatar
Master Guru
@Nilesh

There are 2 unhealthy nodes, if you click on "2" under unhealthy nodes section, you will get reason why they are unhealthy, it could be because of bad disk etc. Please check and try to check nodemanager's logs, you will get more info in there.

View solution in original post

12 REPLIES 12

avatar
Rising Star

Hi, Nilesh. Have you checked the yarn scheduler? Is the default queue out of resource?

avatar
Super Guru

@Nilesh

I will suggest yo revisit your yarn memory configuration once. Seems you might be running out of memory.

Can you please let me know what values you have set for yarn.scheduler.capacity.maximum-am-resource-percent ?

If possible please attach -

1. Yarn RM UI snap

2. yarn-site.xml

3. mapred-site.xml

4. Scheduler snap from RM UI -

http://<RM-Hostname>:8088/cluster/scheduler

avatar
Expert Contributor

@Sagar Shimpi

yarn.scheduler.capacity.maximum-am-resource-percent=0.2

Kindly find attached file for reference.

3415-scheduler-snap.jpg

3418-yarn-memory.jpg

yarn-site.xml

mapred-site.xml

avatar
Super Guru

@Nilesh From the resource manager UI i see there are no "Active Nodes" running and hence the "Total Memory" in UI is displaying 0.

Can you check if your node managers are UP and communicating to RM.

avatar
Master Guru
@Nilesh

There are 2 unhealthy nodes, if you click on "2" under unhealthy nodes section, you will get reason why they are unhealthy, it could be because of bad disk etc. Please check and try to check nodemanager's logs, you will get more info in there.

avatar
Expert Contributor

@Kuldeep Kulkarni and @Sagar Shimpi

Issue has been resolved by changing below parameter in yarn-site.xml

yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage

Previously it was 90% I changed it to 99 Now Job is in running state.

Could you please shed some light on this parameter.

avatar
Master Guru

@Nilesh -

yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage

The maximum percentage of disk space utilization allowed after which a disk is marked as bad. Values can range from 0.0 to 100.0. If the value is greater than or equal to 100, the nodemanager will check for full disk. This applies to yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs.

avatar
Expert Contributor

@Kuldeep Kulkarni

How to find which disk is marked as bad?

avatar
Master Guru

Generally it should be visible on RM UI once you click on the unhealthy nodes.

OR

You can go the unhealthy node(http://<unhealthy-node-manager>:8042/jmx) and check JMX

3443-screen-shot-2016-04-14-at-125436-pm.png