Distributed I/O Benchmark of HDFS

[#1]

hi all

DFSIO is a built-in benchmark tool for HDFS I/O test.

we created the Base Directory to Store Test Results

/benchmarks/TestDFSIO

and when we run the test we get:

 hadoop jar /usr/hdp/2.6.4.0-91/spark2/jars/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar TestDFSIO -write -nrFiles 16 -fileSize 1GB -resFile /tmp/USER-dfsio-write.txt
19/03/31 05:39:57 INFO fs.TestDFSIO: TestDFSIO.1.8
19/03/31 05:39:57 INFO fs.TestDFSIO: nrFiles = 16
19/03/31 05:39:57 INFO fs.TestDFSIO: nrBytes (MB) = 1024.0
19/03/31 05:39:57 INFO fs.TestDFSIO: bufferSize = 1000000
19/03/31 05:39:57 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
19/03/31 05:39:59 INFO fs.TestDFSIO: creating control file: 1073741824 bytes, 16 files
java.io.IOException: Permission denied: user=root, access=WRITE, inode="/benchmarks/TestDFSIO/io_control/in_file_test_io_0":hdfs:hdfs:drwxr-xr-x

where we are wrong?

[#2 · Accepted solution]

@Michael Bronson, Permission issue 🙂 Either run this command with hdfs user or change the ownership of /benchmarks/TestDFSIO to root.

java.io.IOException: Permission denied: user=root, access=WRITE, inode="/benchmarks/TestDFSIO/io_control/in_file_test_io_0":hdfs:hdfs:drwxr-xr-x