Reply
New Contributor
Posts: 1
Registered: ‎05-04-2014

uber mode not activated on CDH5

When launching a  map reduce job for processing 2000 smal pdf files (0.7 Mb) the uver mode does not get activated. See more information below:

 

14/05/04 15:53:00 INFO input.FileInputFormat: Total input paths to process : 2000
14/05/04 15:53:00 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 1, size left: 27516040
14/05/04 15:53:01 INFO mapreduce.JobSubmitter: number of splits:21
14/05/04 15:53:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1399142719676_0002
14/05/04 15:53:03 INFO impl.YarnClientImpl: Submitted application application_1399142719676_0002
14/05/04 15:53:03 INFO mapreduce.Job: The url to track the job: http://localhost.localdomain:8088/proxy/application_1399142719676_0002/
14/05/04 15:53:03 INFO mapreduce.Job: Running job: job_1399142719676_0002
14/05/04 15:54:42 INFO mapreduce.Job: Job job_1399142719676_0002 running in uber mode : false

 

yarn-conf/mapred-site.xml

 

<property><name>mapreduce.job.ubertask.enabled</name><value>true</value></property><property><name>mapreduce.job.ubertask.maxmaps</name><value>100</value></property><property><name>mapreduce.job.ubertask.maxreduces</name><value>0</value></property><property><name>mapreduce.job.ubertask.maxbytes</name><value>167108864</value></property>

 

Any ideas?

 

Many thanks.

Posts: 1,885
Kudos: 423
Solutions: 298
Registered: ‎07-31-2013

Re: uber mode not activated on CDH5

I assume by 0.7 MB you meant that 2000 files totally consume 0.7 MB, and not per file being 0.7 MB?

You may need to additionally ensure the below rules via your configuration:

Memory of AM via (yarn.app.mapreduce.am.resource.mb) must be greater than values of task memory via (mapreduce.map.memory.mb and mapreduce.reduce.memory.mb)

CPU vCores request of AM via (yarn.app.mapreduce.am.resource.cpu-vcores) must be greater than the values of task vCores request via (mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores)
Announcements