Sorry, missed the point that the issue can be reproduced 100%. In CDH5.4, both YARN and MR have some bugs that together in some cases, can cause deadlock or jobs to slow down dramatically.  In your case, it is more likely the latter. Keep us posted on how CDH5.7 behaves. ... View more

Sounds a lot like a deadlock problem in MapReduce. The only solution I can think of is to kill the job and start over. Because the chances of getting into a deadlock are very rare, the job should be able to progress at the second run. The deadlock problem has been seen in 5.7 as well. Until it is completely solved in apache hadoop, it will stay for a while. ... View more

Is it possible for you to try the same settings in your production cluster? If your dev cluster is very close to production server in terms of workload, then I'd guess this will alleviate gc pause in your production cluster as well.  ... View more

http://stackoverflow.com/questions/9257361/how-to-check-heap-size-allocated-for-jvm-by-linux ... View more

Which version of java are you using? I do not know much about GC tuning, but heap size can lead to GC pause sometimes, you can try to increaes the heap size of Node Manager if that is possible. You can also try to collect gc logs by adding java options temporarily "-XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:PrintFLSStatistics=2" ... View more

Jais, for some reason, the configuration is not visiable to me. Can you copy and paste it here as text? ... View more

What is exactly the role that had this warning message? ... View more