I have not been able to pinpoint this problem to CDH 5.5.2 iteself. You could try 5.6 on your test cluster to see if the problem arises again
I'm back again. I upgraded the QA cluster using CDH 5.5.2 with JDK1.8_60 from the default JDK1.7_64 that Cloudera installs. I then tried the test MR job, and it hung again.
But, when I upgraded CDH to CDH5.7.0, the test MR jobs worked! Something has changed back to a working state. So, it looks like that we have to upgrade to CDH5.7.0 instead.
FYI. CDH5.6.0 was not tested. We skipped it because CDH5.6.0 is an unnecessary upgrade.
Does this help?
Thanks a lot for the update. I don't know exactly what have changed between 5.5 and 5.7, but there are definitely a lot of fixes/improvments that might have solved your issues. Will revisit this if we ever see similar issues in the future.