(hadoop-2.0.0-cdh4.5.0)The yarn example runs fail

[#1]

Why there is no no reply?

[#2 · Accepted solution]

@huangxing I have moved this thread to our Yarn discussion board in hopes someone in here can assist you.

Regards,

Clint

[#3]

When I run the example as the document says,there is something wrong,I don't known what happens,someone give me an idea? My resourcemanager and the nodemangers all work well.

[bigdata1 /hadoop/hadoop-2.0.0-cdh4.5.0/share/hadoop/mapreduce]#hadoop jar ./hadoop-mapreduce-examples-2.0.0-cdh4.5.0.jar  grep  /input /output 'dfs[a-z.]+'
14/03/10 19:02:34 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
14/03/10 19:02:34 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
14/03/10 19:02:34 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
14/03/10 19:02:34 INFO input.FileInputFormat: Total input paths to process : 7
14/03/10 19:02:35 INFO mapreduce.JobSubmitter: number of splits:7
14/03/10 19:02:35 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/03/10 19:02:35 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/03/10 19:02:35 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/03/10 19:02:35 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/03/10 19:02:35 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/03/10 19:02:35 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/03/10 19:02:35 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/03/10 19:02:35 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/03/10 19:02:35 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/03/10 19:02:35 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/03/10 19:02:35 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/03/10 19:02:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1394435601283_0011
14/03/10 19:02:35 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
14/03/10 19:02:35 INFO client.YarnClientImpl: Submitted application application_1394435601283_0011 to ResourceManager at bigdata1/192.168.65.66:8032
14/03/10 19:02:35 INFO mapreduce.Job: The url to track the job: http://bigdata1:8088/proxy/application_1394435601283_0011/
14/03/10 19:02:35 INFO mapreduce.Job: Running job: job_1394435601283_0011
14/03/10 19:02:41 INFO mapreduce.Job: Job job_1394435601283_0011 running in uber mode : false
14/03/10 19:02:41 INFO mapreduce.Job:  map 0% reduce 0%
14/03/10 19:02:41 INFO mapreduce.Job: Job job_1394435601283_0011 failed with state FAILED due to: Application application_1394435601283_0011 failed 1 times due to AM Container for appattempt_1394435601283_0011_000001 exited with  exitCode: 1 due to: 
.Failing this attempt.. Failing the application.
14/03/10 19:02:41 INFO mapreduce.Job: Counters: 0
14/03/10 19:02:41 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
14/03/10 19:02:41 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
14/03/10 19:02:42 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
14/03/10 19:02:42 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/root/.staging/job_1394435601283_0012
14/03/10 19:02:42 ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://bigdata1:8020/user/root/grep-temp-1811237380
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://bigdata1:8020/user/root/grep-temp-1811237380
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243)
at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:59)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:468)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:485)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:369)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1287)
at org.apache.hadoop.examples.Grep.run(Grep.java:92)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.Grep.main(Grep.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
[bigdata1 /hadoop/hadoop-2.0.0-cdh4.5.0/share/hadoop/mapreduce]#

[#4 · Accepted solution]

@huangxing I have moved this thread to our Yarn discussion board in hopes someone in here can assist you.

Regards,

Clint

[#5]

Does hdfs://bigdata1:8020/user/root exist? Are you able to run any other examples on this setup?

[#6]

Yes,it exists,and the privilege is 777.The following is from another example,it is the same as the previous one:

[bigdata1 mapreduce]#hadoop jar ./hadoop-mapreduce-examples-2.0.0-cdh4.5.0.jar pi 5 10
Number of Maps = 5
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
14/03/19 21:37:54 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
14/03/19 21:37:54 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
14/03/19 21:37:54 INFO input.FileInputFormat: Total input paths to process : 5
14/03/19 21:37:54 INFO mapreduce.JobSubmitter: number of splits:5
14/03/19 21:37:54 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/03/19 21:37:54 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/03/19 21:37:54 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/03/19 21:37:54 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/03/19 21:37:54 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/03/19 21:37:54 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/03/19 21:37:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1395234357046_0007
14/03/19 21:37:55 INFO client.YarnClientImpl: Submitted application application_1395234357046_0007 to ResourceManager at bigdata1/192.168.65.66:8032
14/03/19 21:37:55 INFO mapreduce.Job: The url to track the job: http://bigdata1:8088/proxy/application_1395234357046_0007/
14/03/19 21:37:55 INFO mapreduce.Job: Running job: job_1395234357046_0007
14/03/19 21:38:09 INFO mapreduce.Job: Job job_1395234357046_0007 running in uber mode : false
14/03/19 21:38:09 INFO mapreduce.Job: map 0% reduce 0%
14/03/19 21:38:09 INFO mapreduce.Job: Job job_1395234357046_0007 failed with state FAILED due to: Application application_1395234357046_0007 failed 1 times due to AM Container for appattempt_1395234357046_0007_000001 exited with exitCode: 1 due to:
.Failing this attempt.. Failing the application.
14/03/19 21:38:09 INFO mapreduce.Job: Counters: 0
Job Finished in 15.584 seconds
java.io.FileNotFoundException: File does not exist: hdfs://bigdata1:8020/user/root/QuasiMonteCarlo_TMP_3_141592654/out/reduce-out
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:824)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

[#7 · Accepted solution]

Your AM died, which is a bit hard to debug. (We're fixing that in CDH5.0.) You can set "yarn.nodemanager.delete.debug-delay-sec" to 6000, which allows the container log to hang around after the failure, so you can find out why the AM died. The container logs are typically in /var/log/hadoop-yarn/container/

It's possible that you have some misconfiguration. If you're not using Cloudera Manager, you should give that a try.

[#8]

I found  the resourcemanager never read configuration from yarn-site.xml  and mapred-site.xml,here is  my configturation:

mapred-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapreduce.jobhistory.address</name>
<value>bigdata1:10020</value>
</property>

<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>bigdata1:19888</value>
</property>
<property>
<name>mapreduce.shuffle.port</name>
<value>8070</value>
</property>

</configuration>

yarn-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>bigdata1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>bigdata1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>bigdata1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>bigdata1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>bigdata1:8088</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR,$YARN_HOME/*,$YARN_HOME/lib/*,$YARN_HOME/share/hadoop/common/*,$YARN_HOME/share/hadoop/yarn,$YARN_HOME/share/hadoop/yarn/lib/*,$YARN_HOME/share/hadoop/mapreduce/*,$YARN_HOME/share/hadoop/mapreduce/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$YARN_HOME/share/hadoop/common/lib/*</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<property>
<name>yarn.dispatcher.exit-on-error</name>
<value>true</value>
</property>

<property>
<name>yarn.nodemanager.local-dirs</name>

<value>/hadoop/hadoop-2.0.0-cdh4.5.0/yarn/${user.name}/nm-local-dir</value>
</property>

<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/hadoop/hadoop-2.0.0-cdh4.5.0/yarn/containers</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/app</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user</value>
</property>

</configuration>

[#9]

I have set "yarn.nodemanager.delete.debug-delay-sec" to 6000,and the container log dir is :

<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/hadoop/hadoop-2.0.0-cdh4.5.0/yarn/containers</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/app</value>
</property>

The dir /hadoop/hadoop-2.0.0-cdh4.5.0/yarn/containers has nothing after  running  the task,I found the configruation in yarn-site.xml never take effect.