Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Map Reduce Jobs not starting on local CDH 5.7.0 installation

Solved Go to solution

Map Reduce Jobs not starting on local CDH 5.7.0 installation

Explorer

Hi experts,

 

I need help, I have installed CDH 5.7.0 on CentOs 6 and all services are up and running.

However testing the installation running the simple pi example and using the following command doesnt execute a map-reduce job:  sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100

The job is planned but nothing is happening...

 

snapshot1.png

 

So the job runs forever.

What can I do to test what is wrong with my installation?!?

Thanks & Regards

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Map Reduce Jobs not starting on local CDH 5.7.0 installation

Explorer

So I've fixed it by adjusting the following Yarn settings:

 

yarn.scheduler.maximum-allocation-mb = 8 GiB
mapreduce.map.memory.mb  = 4 GiB
mapreduce.reduce.memory.mb = 4 GiB

 

And I've got the test example as following command running:

sudo -u hdfs hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100

 

snapshot7.png

 

Thanks for the comments.

6 REPLIES 6

Re: Map Reduce Jobs not starting on local CDH 5.7.0 installation

Expert Contributor

I've seen this happen when the job is requesting more memory than you have allowed YARN to have on any one given host. The job is scheduled, waiting for a node with enough memory to check in and take it, which won't happen since you don't have any one node that can take it on. 

 

Try adjusting mapreduce.map.memory.mb, mapreduce.reduce.memory.mb, and yarn.app.mapreduce.am.resource.mb on your cluster and from the node you are running the job from.

 

here are some helpful resources for YARN tuning:

https://support.pivotal.io/hc/en-us/articles/201462036-MapReduce-YARN-Memory-Parameters

http://www.cloudera.com/documentation/enterprise/5-3-x/topics/cdh_ig_yarn_tuning.html

 

Re: Map Reduce Jobs not starting on local CDH 5.7.0 installation

Explorer

Thanks, I will try it.

The first parameter mapreduce.map.memory.mb was set to 0 GiB, maybe this is the problem.

Re: Map Reduce Jobs not starting on local CDH 5.7.0 installation

Explorer

No success, the job is still running forever :-(

 

I have updated the memory setting from 0 GiB to 1 Gib.

 

snapshot1.png

 

And this memory is also available on the node, but the job will not start.

snapshot2.png

 

I'm lost.

 

snapshot3.png

 

I have not altered the

Re: Map Reduce Jobs not starting on local CDH 5.7.0 installation

Expert Contributor

Based on the screenshot, it looks like you have only one node with 1 GB available?  note that this 1GB is what is usable with both Heap and Virtual combined on the server side, what you specify in mapreduce.map.memory.mb is heap memory only

 

from the link above, re-linked here[1] YARN will automatically calculate the virtual memory to add to the task using the yarn.nodemanager.vmem-pmem-ratio (default 2.1), and then round up to the yarn.scheduler.minimum-allocation-mb (default 1024MiB).

 

the math would be:

1GiB (task heap memory) * 1024 = 1024MiB task heap memory

1024MiB*2.1 = 2150.4 Virtual memory

round to nearest 1024 (2150) = 3072MiB total memory needed for task.

 

If you can't give the cluster more resources, you can tweak the yarn.nodemanager.vmem-pmem-ratio (you do still need some overhead) and you can set yarn.scheduler.minimum-allocation-mb to smaller increments.

setting mapreduce.map.memory.mb and mapreduce.reduce.memory.mb to  (1024MiB/2.1) would allow the AM to run without tweaking either. but note that for YARN, you need the AM to run as well as at least 1 task. so you really must set yarn.scheduler.minimum-allocation-mb to 512 or smaller, and then the target for mapreduce.map.memory.mb and mapreduce.reduce.memory.mb would be 512MiB/2.1 = 240.

 

240 MB may work, depending on what you are running. 

 

[1]https://support.pivotal.io/hc/en-us/articles/201462036-MapReduce-YARN-Memory-Parameters

Re: Map Reduce Jobs not starting on local CDH 5.7.0 installation

Explorer

The "cluster" is pseudo distributed with one node on CentOs6 and I've updated the settings according to your recommendation:

mapreduce.map.memory.mb and mapreduce.reduce.memory.mb = 240 MiB

 

snapshot4.png

 

Deployed the configuration, restarted the serivces but the result is the same, the job runs forever.

 

snapshot5.png

 

I would really need to test something urgently and I'm lost.

The health of the services is good exept "under replicated blocks"

 

snapshot6.png

 

I will follow up on this.

Thanks for any hints.

 

Re: Map Reduce Jobs not starting on local CDH 5.7.0 installation

Explorer

So I've fixed it by adjusting the following Yarn settings:

 

yarn.scheduler.maximum-allocation-mb = 8 GiB
mapreduce.map.memory.mb  = 4 GiB
mapreduce.reduce.memory.mb = 4 GiB

 

And I've got the test example as following command running:

sudo -u hdfs hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100

 

snapshot7.png

 

Thanks for the comments.