Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Re: (hadoop-2.0.0-cdh4.5.0)The yarn example runs fail

avatar
Explorer

Why there is no no reply?

2 ACCEPTED SOLUTIONS

avatar
Guru

@huangxing I have moved this thread to our Yarn discussion board in hopes someone in here can assist you.

 

Regards,

 

Clint

View solution in original post

avatar
Rising Star

Your AM died, which is a bit hard to debug. (We're fixing that in CDH5.0.) You can set "yarn.nodemanager.delete.debug-delay-sec" to 6000, which allows the container log to hang around after the failure, so you can find out why the AM died. The container logs are typically in /var/log/hadoop-yarn/container/

 

It's possible that you have some misconfiguration. If you're not using Cloudera Manager, you should give that a try.

View solution in original post

7 REPLIES 7

avatar
Explorer

When I run the example as the document says,there is something wrong,I don't known what happens,someone give me an idea? My resourcemanager and the nodemangers all work well.

 

[bigdata1 /hadoop/hadoop-2.0.0-cdh4.5.0/share/hadoop/mapreduce]#hadoop jar ./hadoop-mapreduce-examples-2.0.0-cdh4.5.0.jar  grep  /input /output 'dfs[a-z.]+'
14/03/10 19:02:34 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
14/03/10 19:02:34 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
14/03/10 19:02:34 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
14/03/10 19:02:34 INFO input.FileInputFormat: Total input paths to process : 7
14/03/10 19:02:35 INFO mapreduce.JobSubmitter: number of splits:7
14/03/10 19:02:35 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/03/10 19:02:35 WARN conf.Configuration: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/03/10 19:02:35 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/03/10 19:02:35 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/03/10 19:02:35 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/03/10 19:02:35 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/03/10 19:02:35 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/03/10 19:02:35 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/03/10 19:02:35 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/03/10 19:02:35 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/03/10 19:02:35 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/03/10 19:02:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1394435601283_0011
14/03/10 19:02:35 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
14/03/10 19:02:35 INFO client.YarnClientImpl: Submitted application application_1394435601283_0011 to ResourceManager at bigdata1/192.168.65.66:8032
14/03/10 19:02:35 INFO mapreduce.Job: The url to track the job: http://bigdata1:8088/proxy/application_1394435601283_0011/
14/03/10 19:02:35 INFO mapreduce.Job: Running job: job_1394435601283_0011
14/03/10 19:02:41 INFO mapreduce.Job: Job job_1394435601283_0011 running in uber mode : false
14/03/10 19:02:41 INFO mapreduce.Job:  map 0% reduce 0%
14/03/10 19:02:41 INFO mapreduce.Job: Job job_1394435601283_0011 failed with state FAILED due to: Application application_1394435601283_0011 failed 1 times due to AM Container for appattempt_1394435601283_0011_000001 exited with  exitCode: 1 due to: 
.Failing this attempt.. Failing the application.
14/03/10 19:02:41 INFO mapreduce.Job: Counters: 0
14/03/10 19:02:41 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
14/03/10 19:02:41 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
14/03/10 19:02:42 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
14/03/10 19:02:42 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/root/.staging/job_1394435601283_0012
14/03/10 19:02:42 ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://bigdata1:8020/user/root/grep-temp-1811237380
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://bigdata1:8020/user/root/grep-temp-1811237380
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:243)
at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:59)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:269)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:468)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:485)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:369)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1287)
at org.apache.hadoop.examples.Grep.run(Grep.java:92)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.Grep.main(Grep.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
[bigdata1 /hadoop/hadoop-2.0.0-cdh4.5.0/share/hadoop/mapreduce]#

avatar
Guru

@huangxing I have moved this thread to our Yarn discussion board in hopes someone in here can assist you.

 

Regards,

 

Clint

avatar
Rising Star
Does hdfs://bigdata1:8020/user/root exist? Are you able to run any other examples on this setup?

avatar
Explorer

Yes,it exists,and the privilege is 777.The following is from another example,it is the same as the previous one:

 

[bigdata1 mapreduce]#hadoop jar ./hadoop-mapreduce-examples-2.0.0-cdh4.5.0.jar pi 5 10
Number of Maps = 5
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
14/03/19 21:37:54 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
14/03/19 21:37:54 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
14/03/19 21:37:54 INFO input.FileInputFormat: Total input paths to process : 5
14/03/19 21:37:54 INFO mapreduce.JobSubmitter: number of splits:5
14/03/19 21:37:54 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/03/19 21:37:54 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/03/19 21:37:54 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/03/19 21:37:54 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/03/19 21:37:54 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/03/19 21:37:54 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/03/19 21:37:54 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/03/19 21:37:54 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/03/19 21:37:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1395234357046_0007
14/03/19 21:37:55 INFO client.YarnClientImpl: Submitted application application_1395234357046_0007 to ResourceManager at bigdata1/192.168.65.66:8032
14/03/19 21:37:55 INFO mapreduce.Job: The url to track the job: http://bigdata1:8088/proxy/application_1395234357046_0007/
14/03/19 21:37:55 INFO mapreduce.Job: Running job: job_1395234357046_0007
14/03/19 21:38:09 INFO mapreduce.Job: Job job_1395234357046_0007 running in uber mode : false
14/03/19 21:38:09 INFO mapreduce.Job: map 0% reduce 0%
14/03/19 21:38:09 INFO mapreduce.Job: Job job_1395234357046_0007 failed with state FAILED due to: Application application_1395234357046_0007 failed 1 times due to AM Container for appattempt_1395234357046_0007_000001 exited with exitCode: 1 due to:
.Failing this attempt.. Failing the application.
14/03/19 21:38:09 INFO mapreduce.Job: Counters: 0
Job Finished in 15.584 seconds
java.io.FileNotFoundException: File does not exist: hdfs://bigdata1:8020/user/root/QuasiMonteCarlo_TMP_3_141592654/out/reduce-out
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:824)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1704)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

avatar
Rising Star

Your AM died, which is a bit hard to debug. (We're fixing that in CDH5.0.) You can set "yarn.nodemanager.delete.debug-delay-sec" to 6000, which allows the container log to hang around after the failure, so you can find out why the AM died. The container logs are typically in /var/log/hadoop-yarn/container/

 

It's possible that you have some misconfiguration. If you're not using Cloudera Manager, you should give that a try.

avatar
Explorer

I found  the resourcemanager never read configuration from yarn-site.xml  and mapred-site.xml,here is  my configturation:

mapred-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapreduce.jobhistory.address</name>
<value>bigdata1:10020</value>
</property>

<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>bigdata1:19888</value>
</property>
<property>
<name>mapreduce.shuffle.port</name>
<value>8070</value>
</property>

</configuration>

 

 

yarn-site.xml:

 

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>bigdata1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>bigdata1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>bigdata1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>bigdata1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>bigdata1:8088</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR,$YARN_HOME/*,$YARN_HOME/lib/*,$YARN_HOME/share/hadoop/common/*,$YARN_HOME/share/hadoop/yarn,$YARN_HOME/share/hadoop/yarn/lib/*,$YARN_HOME/share/hadoop/mapreduce/*,$YARN_HOME/share/hadoop/mapreduce/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$YARN_HOME/share/hadoop/common/lib/*</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<property>
<name>yarn.dispatcher.exit-on-error</name>
<value>true</value>
</property>

<property>
<name>yarn.nodemanager.local-dirs</name>

<value>/hadoop/hadoop-2.0.0-cdh4.5.0/yarn/${user.name}/nm-local-dir</value>
</property>


<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/hadoop/hadoop-2.0.0-cdh4.5.0/yarn/containers</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/app</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user</value>
</property>

 

</configuration>

 

 

avatar
Explorer

I have set "yarn.nodemanager.delete.debug-delay-sec" to 6000,and the container log dir is :

 

<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/hadoop/hadoop-2.0.0-cdh4.5.0/yarn/containers</value>
</property>
<property>
<description>Where to aggregate logs</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/app</value>
</property>

 

The dir /hadoop/hadoop-2.0.0-cdh4.5.0/yarn/containers has nothing after  running  the task,I found the configruation in yarn-site.xml never take effect.