Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

yarn configure to use all ram

yarn configure to use all ram

New Contributor

we have multiple systems with different cpu and ram each

the cpu can be used to utilize 100%

but i could not do it for the ram

HDP-2.6.4.0

11 REPLIES 11

Re: yarn configure to use all ram

Expert Contributor
@Ilia K

You can manually configure your memory allocation settings based on the hardware specifications. Please refer below links for more tuning up RAM in yarn configuration.

https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_command-line-installation/content/determ...

Re: yarn configure to use all ram

New Contributor

i want to configure it via the ambari configuraion manager, so theat each host will use 100% ram and CPU

Re: yarn configure to use all ram

Expert Contributor

For changing memory value through ambari configuration manager, login into ambari UI,

Yarn —> Configs —> Settings —> memory —> Memory allocated for all YARN containers on a node and Minimum / Maximum Container Size memory.

and for advanced turning of yarn modify

Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site

Re: yarn configure to use all ram

New Contributor

i looking for an option to set 100% and not to set fixd ram or cpu

Re: yarn configure to use all ram

Expert Contributor

If you set 100% ram only for yarn then other services will get crash / GC issue. It's not recommendable to set all physical memory to one component.

Re: yarn configure to use all ram

New Contributor

ok, then configure a 90% mark, the main issue that i want to manage single configuraion file for all hardware

Re: yarn configure to use all ram

Expert Contributor

For example your physical memory size is 10GB you need to allocated for every container request at the RM, is around 9GB (90% of your physical memory) then you can mention in below parameter

yarn.scheduler.maximum-allocation-mb 9216

Similarly amount of physical memory allocated for containers in node manager, that can be mention using below parameter.

yarn.nodemanager.resource.memory-mb 9216

These all settings are manage by single configuration file I.e., yarn-site.xml which you can modify via ambari under Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site

I hope that the above answers your questions. Please accept the answer you found most useful.

Re: yarn configure to use all ram

New Contributor

i know to to configure it,

for example now i have 3 types of hardares:

3 computers with 12 core cpu and 64gb ram

6 computers with 16 core and 64gb ram

and 4 computers with 48 core and 256gb ram

and i dont want to manage 3 configuraion files , i want to manage one, for instance each computer will allocate all cpus and 90% of ram for yarn tasks

Re: yarn configure to use all ram

Expert Contributor

For your question, answer is No. As of now we can't mange memory of individual node level using percentage. Currently yarn support only CPU, using below configuration.

yarn.nodemanager.resource.percentage-physical-cpu-limit100Percentage of CPU that can be allocated for containers. This setting allows users to limit the amount of CPU that YARN containers use. Currently functional only on Linux using cgroups. The default is to use 100% of CPU.