Reply
Champion Alumni
Posts: 196
Registered: ‎11-18-2014

yarn - the job got stock

[ Edited ]

Hello,

 

 

This is the second time that this happens to me..

 

I have a cron job that is computing sessions. 

If in the same time, I am running some other jobs, 

from time to time the cron job is arriving in a state where

- It has only one map to do

- It already started the reducers

- The workers are doing nothing (CPU is very low, and memory is normal)

 

I waited twice the time the job would normally finished, and it didn't advanced,

even though stopped doing other queries on the cluster.

 

I run afterwards, the cron, (without doing anything on the cluster) and it finished in the normal amount of time (1h30)

 

How should I configure yarn in order to make him stop arriving in this state?

Why is this happening?

 

Thank you

 

GHERMAN Alina
Highlighted
Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Re: yarn - the job got stock

I think I found the reason why this is happening:

 

http://stackoverflow.com/questions/11672676/when-do-reduce-tasks-start-in-hadoop 

 

By default mapreduce.job.reduce.slowstart.completedmaps is 0.8.

 

Thank you!

GHERMAN Alina
Announcements