The issue could be associated with the corrupt permissions in NM local-dir (usercache) are causing the shuffle fetch failure which may occur due to the below reasons, 1.Disk failure cause the fetch failure 2. Network issue causes the fetch failure This can be overcome by carrying out the below steps in the NodeManger installed nodes. * Stop the NM instance on the node * Remove YARN NM local-dir path - sudo rm -rf /JBOD_D${i}/hadoop/cdh/yarn/nm/*" where ${i} is the iteration over the disks that each participate as NM local-dir storage - the idea is to leave /JBOD_D${i}/hadoop/cdh/yarn/nm in place on each disk, but remove everything below that point. * Restart the NM instance and it will recreate the structure with the appropriate permissions. Once you have implemented the above solution and if it continues to working well, then please spare few minutes to hit the "Accept as Solution"
... View more
