Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hadoop mapreduce gpfs

Highlighted

hadoop mapreduce gpfs

Super Collaborator

Hi:

Iam trying execute the hortonworks example to gpfs and i am receiving this error:

java.io.FileNotFoundException: File does not exist: gpfs:/user/hdfs/QuasiMonteCarlo_1468247525413_1210900368/out/reduce-out
[hdfs@xxxxx src]$ hadoop jar /usr/hdp/2.4.2.0-258/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar pi 10 10
WARNING: Use "yarn jar" to launch YARN applications.
Number of Maps  = 10
Samples per Map = 10
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #0
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #1
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #2
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #3
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #4
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #5
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #6
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #7
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #8
16/07/11 16:32:06 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
Wrote input for Map #9
Starting Job
16/07/11 16:32:07 INFO impl.TimelineClientImpl: Timeline service address: http://lnxnpi01.risa:8188/ws/v1/timeline/
16/07/11 16:32:07 INFO client.RMProxy: Connecting to ResourceManager at lnxnpi01.risa/xxxxxx:8050
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of range (1..3): 10; replication set to 2
16/07/11 16:32:07 INFO input.FileInputFormat: Total input paths to process : 10
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of range (1..3): 10; replication set to 2
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
16/07/11 16:32:07 INFO mapreduce.JobSubmitter: number of splits:10
16/07/11 16:32:07 WARN gpfs.GeneralParallelFileSystem: replication out of  file system max range : 3; replication set to 2
16/07/11 16:32:08 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468244627042_0013
16/07/11 16:32:08 INFO impl.YarnClientImpl: Submitted application application_1468244627042_0013
16/07/11 16:32:08 INFO mapreduce.Job: The url to track the job: http://lnxnpi01.risa:8088/proxy/application_1468244627042_0013/
16/07/11 16:32:08 INFO mapreduce.Job: Running job: job_1468244627042_0013
16/07/11 16:32:10 INFO mapreduce.Job: Job job_1468244627042_0013 running in uber mode : false
16/07/11 16:32:10 INFO mapreduce.Job:  map 0% reduce 0%
16/07/11 16:32:10 INFO mapreduce.Job: Job job_1468244627042_0013 failed with state FAILED due to: Application application_1468244627042_0013 failed 2 times due to AM Container for appattempt_1468244627042_0013_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://xxxxxxx:8088/cluster/app/application_1468244627042_0013Then, click on links to logs of each attempt.
Diagnostics: java.io.IOException: GPFSC00023E: Unable to establish communication with file system
Failing this attempt. Failing the application.
16/07/11 16:32:10 INFO mapreduce.Job: Counters: 0
Job Finished in 3.77 seconds
java.io.FileNotFoundException: File does not exist: gpfs:/user/hdfs/QuasiMonteCarlo_1468247525413_1210900368/out/reduce-out
        at org.apache.hadoop.fs.gpfs.GeneralParallelFileSystem.getFileStatus(GeneralParallelFileSystem.java:881)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1752)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1776)
        at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
        at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

2 REPLIES 2

Re: hadoop mapreduce gpfs

Super Guru

@Roberto Sancho can you provide link to example you are following?

Don't have an account?
Coming from Hortonworks? Activate your account here