Moving to Yarn from Mapreduce-0.20

[#1]

 Hi,

   I am using cloudera 4.4.0 VM. I am planing to move from mapreduce 0.20 to Yarn.I have done the following.

1. Stopped Mapreduce service

2. start yarn service

3. sudo alternatives --set hadoop-conf  /etc/hadoop/conf.cloudera.yarn1

With this when i run the example wordcount job

yarn jar  hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar wordcount /user/cloudera/input/core-site.xml /tmp/out1

The job is submitted but the mappers are always in pending step and never move forward.There are no error in the application manager log.

The output is copied here

[cloudera@localhost hadoop-mapreduce]$ yarn jar hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar wordcount hdfs://localhost:8020/user/cloudera/input/core-site.xml /tmp/out1
2014-03-18 03:17:36,619 INFO [main] service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
2014-03-18 03:17:37,157 INFO [main] service.AbstractService(AbstractService.java:start(94)) -Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
2014-03-18 03:17:41,504 INFO [main] input.FileInputFormat(FileInputFormat.java:listStatus(245)) - Total input paths to process : 1
2014-03-18 03:17:42,118 INFO [main] mapreduce.JobSubmitter(JobSubmitter.java:submitJobInternal(368)) - number of splits:1
2014-03-18 03:17:42,286 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapred.jar is deprecated.
Instead, use mapreduce.job.jar
2014-03-18 03:17:42,304 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapred.output.value.class is
deprecated. Instead, use mapreduce.job.output.value.class
2014-03-18 03:17:42,316 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapreduce.combine.class is
deprecated. Instead, use mapreduce.job.combine.class
2014-03-18 03:17:42,322 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapreduce.map.class is
deprecated. Instead, use mapreduce.job.map.class
2014-03-18 03:17:42,327 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapred.job.name is
deprecated. Instead, use mapreduce.job.name
2014-03-18 03:17:42,329 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapreduce.reduce.class is
deprecated. Instead, use mapreduce.job.reduce.class
2014-03-18 03:17:42,334 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapred.input.dir is
deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
2014-03-18 03:17:42,334 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapred.output.dir is
deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
2014-03-18 03:17:42,339 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapred.map.tasks is
deprecated. Instead, use mapreduce.job.maps
2014-03-18 03:17:42,346 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapred.output.key.class is
deprecated. Instead, use mapreduce.job.output.key.class
2014-03-18 03:17:42,352 WARN [main] conf.Configuration(Configuration.java:warnOnceIfDeprecated(824)) - mapred.working.dir is
deprecated. Instead, use mapreduce.job.working.dir
2014-03-18 03:17:45,166 INFO [main] mapreduce.JobSubmitter(JobSubmitter.java:printTokens(438)) - Submitting tokens for job:
job_1395137802361_0001
2014-03-18 03:17:50,030 INFO [main] client.YarnClientImpl(YarnClientImpl.java:submitApplication(124)) - Submitted application
application_1395137802361_0001 to ResourceManager at
localhost.localdomain/127.0.0.1:8032
2014-03-18 03:17:51,237 INFO [main] mapreduce.Job (Job.java:submit(1273)) -The url to track the job:
http://localhost.localdomain:8088/proxy/application_1395137802361_0001/
2014-03-18 03:17:51,259 INFO [main] mapreduce.Job(Job.java:monitorAndPrintJob(1318)) - Running job: job_1395137802361_0001
2014-03-18 03:18:19,971 INFO [main] mapreduce.Job(Job.java:monitorAndPrintJob(1339)) - Job job_1395137802361_0001 running in
uber mode : false
2014-03-18 03:18:19,999 INFO [main] mapreduce.Job(Job.java:monitorAndPrintJob(1346)) - map 0% reduce 0%

This is where it gets stuck.

[#2 · Accepted solution]

This may happen if your YARN service is misconfigured w.r.t. memory resources. An MR job requires 1.5g for its AppMaster, and subsequently 1g for each task. Therefore, NodeManagers should be publishing at least 2.5g or 3g for MR jobs to run.

Can you ensure your value in CM -> YARN -> Configuration, for field
"yarn.nodemanager.resource.memory-mb" is set to at least 3 GB or higher? If it isn't, please set it appropriately as per your machine's ability, and restart the YARN service and retry the job submission.

You can also alternatively reduce the default memory demands via client-side properties yarn.app.mapreduce.am.resource.mb (for the MR AppMaster) and set of mapreduce.map.memory.mb/mapreduce.reduce.memory.mb (for the Tasks). These properties accept integer values in MB units.

[#3]

Thanks for your prompt reply