Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can we allocate more resources manually to the running job in yarn or mapreduce?

Highlighted

Can we allocate more resources manually to the running job in yarn or mapreduce?

Guru

If we have submitted one job and later we want to increase some resources to that jobs so can we do that ?

I found following thread was opened long back but checking if we some way to achieve this .

https://issues.apache.org/jira/browse/YARN-1197

2 REPLIES 2

Re: Can we allocate more resources manually to the running job in yarn or mapreduce?

@Saurabh

There are certain things which we reset when executing a job. If it is in Hive then the following may work

set mapreduce.map.memory.mb=9000;

set mapreduce.map.java.opts=-Xmx7200m;

set mapreduce.reduce.memory.mb=9000;

set mapreduce.reduce.java.opts=-Xmx7200m;

But if you are talking about resource allocation like increasing the size of a queue then I dont think that achievable on the fly when executing a job. Below link might help

https://community.hortonworks.com/content/supportkb/48788/i-am-seeing-outofmemory-errors-when-i-run-...

Re: Can we allocate more resources manually to the running job in yarn or mapreduce?

New Contributor

You can use the queue setting to move your application from low resource queue to high resource queue while the job is running. Use "yarn application -movetoqueue" command. Also you can use the ambari to allocate high % of resource to any given queue.

Don't have an account?
Coming from Hortonworks? Activate your account here