Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hadoop mapreduce gpfs

hadoop mapreduce gpfs

Super Collaborator

Hi:

Iam trying execute the hortonworks example to gpfs and i am receiving this error:

java.io.FileNotFoundException: File does not exist: gpfs:/user/hdfs/QuasiMonteCarlo_1468247525413_1210900368/out/reduce-out
[hdfs@xxxxx src]$ hadoop jar /usr/hdp/2.4.2.0-258/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar pi 10 10
WARNING: Use "yarn jar" to launch YARN applications.
Number of Maps  = 10
Samples per Map = 10
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #0
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #1
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #2
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #3
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #4
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #5
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #6
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #7
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #8
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #9
Starting Job
16/07/11 16:32:07 INFO impl.TimelineClientImpl: Timeline service address: http://lnxnpi01.risa:8188/ws/v1/timeline/
16/07/11 16:32:07 INFO client.RMProxy: Connecting to ResourceManager at lnxnpi01.risa/xxxxxx:8050
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of range (1..3): 10; replication set to 2
16/07/11 16:32:07 INFO input.FileInputFormat: Total input paths to process : 10
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of range (1..3): 10; replication set to 2
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
16/07/11 16:32:07 INFO mapreduce.JobSubmitter: number of splits:10
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
16/07/11 16:32:08 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468244627042_0013
16/07/11 16:32:08 INFO impl.YarnClientImpl: Submitted application application_1468244627042_0013
16/07/11 16:32:08 INFO mapreduce.Job: The url to track the job: http://lnxnpi01.risa:8088/proxy/application_1468244627042_0013/
16/07/11 16:32:08 INFO mapreduce.Job: Running job: job_1468244627042_0013
16/07/11 16:32:10 INFO mapreduce.Job: Job job_1468244627042_0013 running in uber mode : false
16/07/11 16:32:10 INFO mapreduce.Job:  map 0% reduce 0%
16/07/11 16:32:10 INFO mapreduce.Job: Job job_1468244627042_0013 failed with state FAILED due to: Application application_1468244627042_0013 failed 2 times due to AM Container for appattempt_1468244627042_0013_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://xxxxxxx:8088/cluster/app/application_1468244627042_0013Then, click on links to logs of each attempt.
Diagnostics: java.io.IOException: GPFSC00023E: Unable to establish communication with file system
Failing this attempt. Failing the application.
16/07/11 16:32:10 INFO mapreduce.Job: Counters: 0
Job Finished in 3.77 seconds
java.io.FileNotFoundException: File does not exist: gpfs:/user/hdfs/QuasiMonteCarlo_1468247525413_1210900368/out/reduce-out
        at org.apache.hadoop.fs.gpfs.GeneralParallelFileSystem.getFileStatus(GeneralParallelFileSystem.java:881)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1752)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1776)
        at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
        at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

2 REPLIES 2
Highlighted

Re: hadoop mapreduce gpfs

Super Guru

@Roberto Sancho can you provide link to example you are following?

Don't have an account?
Coming from Hortonworks? Activate your account here