01-22-2018 07:43 AM - last edited on 01-22-2018 08:21 AM by cjervis
I am running a well-known benchmark test to measure the I/O of my cluster using the hadoop-test-2.6.0-mr1-cdh5.13.0.jar using the TestDFSIO test. Normally the ouput should be written to TestDFSIO_results.log in the current folder of the bash (not hdfs) where I run the command. THis is in my case just my home folder.
I am running the command as follows:
sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/jars/hadoop-test-2.6.0-mr1-cdh5.13.0.jar TestDFSIO -zrite -nrFiles 2 -size 100MB.
As it seems I need to run the job with the hdfs user, I have changed the permission of the default output file to 777 and set the owner and group to hdfs, but also this does not help (also previously when my own user was the owner it did not work).
Thanks for any help!!
02-06-2018 02:32 AM
I just solved this issue by starting the command while I was in /tmp. The problem was the user of the hadoop jar job (hdfs) did not have sufficient rights to write to the local system. When executing from /tmp/ the output file with the results was correctly produced.
Hope this can help others in the future!