Support Questions

Find answers, ask questions, and share your expertise

java.io.FileNotFoundException: File does not exist: hdfs://ABC.xyz.net:8020/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.2-mr-framework.tar.gz

avatar

I was testing if mapreduce is working properly on my cloudera 6.3, CDH 6.3.2 cluster.

I have installed cloudera using one-click-installer Proof-of-concept one!

 

while i excecuted the below command as it is in this link

https://docs.cloudera.com/documentation/other/tutorial/CDH5/topics/ht_usage.html

 

 

Command:

sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100

 

i Got below error:

 

WARNING: Use "yarn jar" to launch YARN applications.
Number of Maps = 10
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
20/01/02 15:18:01 INFO client.RMProxy: Connecting to ResourceManager at ABC.xyz.net/192.168.xx.xx:8032
java.io.FileNotFoundException: File does not exist: hdfs://ABC.xyz.net:8020/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.2-mr-framework.tar.gz
at org.apache.hadoop.fs.Hdfs.getFileStatus(Hdfs.java:145)
at org.apache.hadoop.fs.AbstractFileSystem.resolvePath(AbstractFileSystem.java:488)
at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2225)
at org.apache.hadoop.fs.FileContext$25.next(FileContext.java:2221)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.resolve(FileContext.java:2227)
at org.apache.hadoop.fs.FileContext.resolvePath(FileContext.java:607)
at org.apache.hadoop.mapreduce.JobSubmitter.addMRFrameworkToDistributedCache(JobSubmitter.java:460)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:146)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:307)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:360)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:368)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 

 

Please help me understand the issue.

Thanks for the reply.
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
at org.apache.hadoop.util.RunJar.main(RunJar.java:227)

2 ACCEPTED SOLUTIONS

avatar
Super Guru
@backlashhardik ,

Please try below:

1. update HDFS path permissions to make sure:

hdfs dfs -chmod 755 /user/yarn
hdfs dfs -chmod 755 /user/yarn/mapreduce

you might need to run as "hdfs" user

2. Then install the Yarn MapReduce Frame jar
CM -> YARN -> Actions -> Install YARN MapReduce frame jar

Please let me know if that helps.

Cheers
Eric

View solution in original post

avatar
Super Guru
Sorry, forgot to mention that you might need to stop YARN service first before step 2.

Cheers
Eric

View solution in original post

9 REPLIES 9

avatar
Super Guru
@backlashhardik ,

Please try below:

1. update HDFS path permissions to make sure:

hdfs dfs -chmod 755 /user/yarn
hdfs dfs -chmod 755 /user/yarn/mapreduce

you might need to run as "hdfs" user

2. Then install the Yarn MapReduce Frame jar
CM -> YARN -> Actions -> Install YARN MapReduce frame jar

Please let me know if that helps.

Cheers
Eric

avatar
Explorer

It worked for me.

Thanks

avatar
New Contributor

solution worked for me. Thank you

avatar
New Contributor

Thank you.

 

Solution worked for me.

avatar
New Contributor

hi i have similiar problem but it is kafka  instead of mapr. would you mind to check kafka to hdfs file not found exception 

avatar
Super Guru
Sorry, forgot to mention that you might need to stop YARN service first before step 2.

Cheers
Eric

avatar

WORKED FOR ME PREFECTLY!

Thanks for identifying it so soon!

@EricL 🙂

avatar
Super Guru
No problem, glad that it helped.

Cheers
Eric

avatar
Explorer

Well done! it worked for me.

------------------------------------

and from my test... 

just this step CM -> YARN -> Actions -> Install YARN MapReduce frame jar  enough.

nothing else. CM will do else for u automatically.

 

from CDH 6.2