Support Questions
Find answers, ask questions, and share your expertise

Name node and Data node throughput in Federated cluster

Name node and Data node throughput in Federated cluster

Hi,

I have setup a HDFS Federation with HA. Now I am trying to run DFSIO throughput tests but facing an issue as shown below.

What could be the possible cause for this ?

[root@hadoop4ind mapreduce]# hadoop jar hadoop-*tests*.jar TestDFSIO -write -nrFiles 10000 -fileSize 1000
16/11/16 16:17:16 INFO fs.TestDFSIO: TestDFSIO.1.8
16/11/16 16:17:16 INFO fs.TestDFSIO: nrFiles = 10000
16/11/16 16:17:16 INFO fs.TestDFSIO: nrBytes (MB) = 1000.0
16/11/16 16:17:16 INFO fs.TestDFSIO: bufferSize = 1000000
16/11/16 16:17:16 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/11/16 16:17:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/16 16:17:18 INFO fs.TestDFSIO: creating control file: 1048576000 bytes, 10000 files
java.io.FileNotFoundException: /benchmarks
        at org.apache.hadoop.fs.viewfs.InodeTree.resolve(InodeTree.java:402)
        at org.apache.hadoop.fs.viewfs.ViewFileSystem.delete(ViewFileSystem.java:310)
        at org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:304)
        at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:815)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:712)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:130)
        at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:138)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
1 REPLY 1

Re: Name node and Data node throughput in Federated cluster

Hi @Viraj Vekaria Can you make sure the user who is running the job has permission to write into respective directory ?

Pls refer - https://community.hortonworks.com/questions/49758/testdfsio-output-error.html

Let me know if that helps.