Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

yarn - the job got stock

yarn - the job got stock

Champion Alumni

Hello,

 

 

This is the second time that this happens to me..

 

I have a cron job that is computing sessions. 

If in the same time, I am running some other jobs, 

from time to time the cron job is arriving in a state where

- It has only one map to do

- It already started the reducers

- The workers are doing nothing (CPU is very low, and memory is normal)

 

I waited twice the time the job would normally finished, and it didn't advanced,

even though stopped doing other queries on the cluster.

 

I run afterwards, the cron, (without doing anything on the cluster) and it finished in the normal amount of time (1h30)

 

How should I configure yarn in order to make him stop arriving in this state?

Why is this happening?

 

Thank you

 

GHERMAN Alina
1 REPLY 1

Re: yarn - the job got stock

Champion Alumni

I think I found the reason why this is happening:

 

http://stackoverflow.com/questions/11672676/when-do-reduce-tasks-start-in-hadoop 

 

By default mapreduce.job.reduce.slowstart.completedmaps is 0.8.

 

Thank you!

GHERMAN Alina