question Re: How to override 'mapred.user.jobconf.limit' value in driver class with value set in mapred-site. in Support Questions

How to override 'mapred.user.jobconf.limit' value in driver class with value set in mapred-site.xml?

vganji — Fri, 16 Sep 2022 09:30:13 GMT

I have mapreduce job which requires more than current value of 10mb which is set in mapred-site.xml. I want to use temporarily increased value for my current job. Can I set it in the driver class, like 'job.setXXXXXX()' ? And also I dont want to use -D option at command line. Any suggestions ?

Re: How to override 'mapred.user.jobconf.limit' value in driver class with value set in mapred-site.

Harsh J — Thu, 28 May 2015 04:20:49 GMT

If this is an MR1 question, then the config mentioned is a server-side
limit, and cannot be overridden at a per-job level. You will need to
raise it on the JobTracker config and restart the JobTracker to set it
into effect.

Re: How to override 'mapred.user.jobconf.limit' value in driver class with value set in mapred-site.

vganji — Thu, 28 May 2015 04:32:26 GMT

I know that it could be set in jobtracker but, I am trying to set it at job level to avoid jobtracker restart. So, as I understand it is not at all possible to set it at job level ? Interesting. Thanks Harsh.

Re: How to override 'mapred.user.jobconf.limit' value in driver class with value set in mapred-site.

Harsh J — Thu, 28 May 2015 04:36:49 GMT

Yes, it is an administrative limit, so it cannot be changed per job.

Re: How to override 'mapred.user.jobconf.limit' value in driver class with value set in mapred-site.

vganji — Thu, 28 May 2015 04:40:57 GMT

Thanks Harsh, appreciate it.