Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how does Dynamic Resource Pools in CM works? which config files are changed by this resource tool?

SOLVED Go to solution

how does Dynamic Resource Pools in CM works? which config files are changed by this resource tool?

New Contributor

and why there is no capacity-scheduler.xml in yarn's conf folder? as described in http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

 

Thanks

2 ACCEPTED SOLUTIONS

Accepted Solutions

Re: how does Dynamic Resource Pools in CM works? which config files are changed by this resource too

See documentation here:

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Managing-...

 

Dynamic Resource Pools uses the Fair Scheduler, not the Capacity Scheduler. So you don't need capacity-scheduler.xml.

 

Also keep in mind that cloudera doesn't generally put server-only configuration into /etc/hadoop/conf. You can view a particular role's configuration files in the CM UI on the Processes tab for that role (click Show to unhide them). These will appear in a directory in /var/run/cloudera-scm-agent/. You can see the directory name in the stderr log on the Processes tab.

 

Thanks,

Darren

Re: how does Dynamic Resource Pools in CM works? which config files are changed by this resource too

New Contributor

thanks, i found the config files that are changed

 

they are in

/var/run/cloudera-scm-agent/process/XXX-yarn-RESOURCEMANAGER-refresh

on yarn RM node

and

/var/run/cloudera-scm-agent/process/XXX-impala-IMPALAD/impala-conf/fair-scheduler.xml

/var/run/cloudera-scm-agent/process/XXX-impala-IMPALAD/impala-conf/llama-site.xml

on impalad node

4 REPLIES 4

Re: how does Dynamic Resource Pools in CM works? which config files are changed by this resource too

See documentation here:

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Managing-...

 

Dynamic Resource Pools uses the Fair Scheduler, not the Capacity Scheduler. So you don't need capacity-scheduler.xml.

 

Also keep in mind that cloudera doesn't generally put server-only configuration into /etc/hadoop/conf. You can view a particular role's configuration files in the CM UI on the Processes tab for that role (click Show to unhide them). These will appear in a directory in /var/run/cloudera-scm-agent/. You can see the directory name in the stderr log on the Processes tab.

 

Thanks,

Darren

Re: how does Dynamic Resource Pools in CM works? which config files are changed by this resource too

New Contributor

thanks, i found the config files that are changed

 

they are in

/var/run/cloudera-scm-agent/process/XXX-yarn-RESOURCEMANAGER-refresh

on yarn RM node

and

/var/run/cloudera-scm-agent/process/XXX-impala-IMPALAD/impala-conf/fair-scheduler.xml

/var/run/cloudera-scm-agent/process/XXX-impala-IMPALAD/impala-conf/llama-site.xml

on impalad node

Is it possible, an YARN DRP has priority against other pools?

Explorer

Is it possible, an YARN DRP has priority against other pools?

ie. If i already have 2 DRP created

DRP1 = 500Gb
DRP2 = 200Gb

and i want to add new one DRP3 = 700Gb that have priority against other pools and use all available resources are directed to DRP3. Is this possible?

Re: Is it possible, an YARN DRP has priority against other pools?

Cloudera Employee

You can try to increase the weight of the DRP3, so it will get more priority and jobs submitted to this pool will get more resources than other pools based on the weight configured.