05-03-2018 12:05 PM - last edited on 05-04-2018 06:02 AM by cjervis
We are seeing some issues with or MR job and one reducer never completes. it keeps running for 8-9 hrs. we initially tried increasing the reducer memory and increased number of reducers from 40 to 60.
we noticed that Reduce Output Records are increasing at a slow pace. noting is being written to disk.
Out of the 60 reducers reducer # 32 is taking longer. the rest of the reducers complete in around 1 min. we suspect its a data skew issue. How can i further troubleshoot this issue ? or has anyone seen this in the past ?
05-06-2018 11:44 PM