I am trying to run a benchmark job, with the following command : yarn jar /path/to/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 1000 -resFile /tmp/TESTDFSio.txt but my job fails with following error messages :
16/01/22 15:08:47 INFO mapreduce.Job: Task Id : attempt_1453395961197_0017_m_000008_2, Status : FAILED Application application_1453395961197_0017 initialization failed (exitCode=255) with output: main : command provided 0
main : user is foo
main : requested yarn user is foo
Path /mnt/sdb1/yarn/local/usercache/foo/appcache/application_1453395961197_0017 has permission 700 but needs permission 750.
Path /var/hadoop/yarn/local/usercache/foo/appcache/application_1453395961197_0017 has permission 700 but needs permission 750. Did not create any app directories
Even when I change these directories permission to 750, I get errors. Also these caches dont get cleaned off, after one job'and create collisons when running the next job. Any insights ?
@Anilkumar Panda can you run service checks for Mapreduce2, YARN and HDFS. You should restart YARN service and it will change permissions as necessary unless there are other issues, in that case, we need to check umask and mount options on your disk.
Have tried that, also the issue is, when a new folder is created, the permissions dont apply, hence the job starts failing.
Some cleaning up is not happening correctly, but I am unable to locate the issue 😞
Let's try this
check for yarn.nodemanager.local-dirs
for user foo delete everything under usercache for the user in all data nodes.
[root@phdns02 conf]# ls -l /hadoop/yarn/local/usercache/foo/
drwxr-xr-x. 2 yarn hadoop 4096 Jan 23 15:06 appcache
drwxr-xr-x. 2 yarn hadoop 4096 Jan 23 14:01 filecache
@Artem Ervits @Neeraj SabharwalI have noticed a few conflicting settings in the Yarn site.xml.yarn.nodemanager.container-executor.class = org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutorand we dont have same linux users across the cluster. Hence waiting for the users to be created. Once that is done will test and post the result.