Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Name node and Data node throughput in Federated cluster

Highlighted

Name node and Data node throughput in Federated cluster

New Contributor

Hi,

I have setup a HDFS Federation with HA. Now I am trying to run DFSIO throughput tests but facing an issue as shown below.

What could be the possible cause for this ?

[root@hadoop4ind mapreduce]# hadoop jar hadoop-*tests*.jar TestDFSIO -write -nrFiles 10000 -fileSize 1000
16/11/16 16:17:16 INFO fs.TestDFSIO: TestDFSIO.1.8
16/11/16 16:17:16 INFO fs.TestDFSIO: nrFiles = 10000
16/11/16 16:17:16 INFO fs.TestDFSIO: nrBytes (MB) = 1000.0
16/11/16 16:17:16 INFO fs.TestDFSIO: bufferSize = 1000000
16/11/16 16:17:16 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/11/16 16:17:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/16 16:17:18 INFO fs.TestDFSIO: creating control file: 1048576000 bytes, 10000 files
java.io.FileNotFoundException: /benchmarks
        at org.apache.hadoop.fs.viewfs.InodeTree.resolve(InodeTree.java:402)
        at org.apache.hadoop.fs.viewfs.ViewFileSystem.delete(ViewFileSystem.java:310)
        at org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:304)
        at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:815)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:712)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:130)
        at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:138)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
1 REPLY 1

Re: Name node and Data node throughput in Federated cluster

Hi @Viraj Vekaria Can you make sure the user who is running the job has permission to write into respective directory ?

Pls refer - https://community.hortonworks.com/questions/49758/testdfsio-output-error.html

Let me know if that helps.

Don't have an account?
Coming from Hortonworks? Activate your account here