Today our Cloudera CDH 5.4 cluster throw below error when we try to submit our job from hdfs user
15/10/14 20:08:17 INFO mapreduce.Job: Job job_1444829272447_0004 failed with state FAILED due to: Application application_1444829272447_0004 failed 2 times due to AM Container for appattempt_1444829272447_0004_000002 exited with exitCode: -1000 For more detailed output, check application tracking page:http://node1.pixilapps.com:8088/proxy/application_1444829272447_0004/Then, click on links to logs of each attempt. Diagnostics: Not able to initialize user directories in any of the configured local directories for user hdfs Failing this attempt. Failing the application.
I made some research defining the problem. main reason of the problem is deletion of local directories on YARN startup, but in practice it fails to delete the directories because of permission problems. The top-level usercache directory is owned by the user but is in a directory that is not writable by the user. But I am not getting actual error about it