yarn configure to use all ram

[#1]

we have multiple systems with different cpu and ram each

the cpu can be used to utilize 100%

but i could not do it for the ram

HDP-2.6.4.0

[#2]

@Ilia K

You can manually configure your memory allocation settings based on the hardware specifications. Please refer below links for more tuning up RAM in yarn configuration.

https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_command-line-installation/content/determ...

[#3]

i want to configure it via the ambari configuraion manager, so theat each host will use 100% ram and CPU

[#4]

For changing memory value through ambari configuration manager, login into ambari UI,

Yarn —> Configs —> Settings —> memory —> Memory allocated for all YARN containers on a node and Minimum / Maximum Container Size memory.

and for advanced turning of yarn modify

Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site

[#5]

i looking for an option to set 100% and not to set fixd ram or cpu

[#6]

If you set 100% ram only for yarn then other services will get crash / GC issue. It's not recommendable to set all physical memory to one component.

[#7]

ok, then configure a 90% mark, the main issue that i want to manage single configuraion file for all hardware

[#8]

For example your physical memory size is 10GB you need to allocated for every container request at the RM, is around 9GB (90% of your physical memory) then you can mention in below parameter

yarn.scheduler.maximum-allocation-mb 9216

Similarly amount of physical memory allocated for containers in node manager, that can be mention using below parameter.

yarn.nodemanager.resource.memory-mb 9216

These all settings are manage by single configuration file I.e., yarn-site.xml which you can modify via ambari under Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site

I hope that the above answers your questions. Please accept the answer you found most useful.

[#9]

i know to to configure it,

for example now i have 3 types of hardares:

3 computers with 12 core cpu and 64gb ram

6 computers with 16 core and 64gb ram

and 4 computers with 48 core and 256gb ram

and i dont want to manage 3 configuraion files , i want to manage one, for instance each computer will allocate all cpus and 90% of ram for yarn tasks

[#10]

For your question, answer is No. As of now we can't mange memory of individual node level using percentage. Currently yarn support only CPU, using below configuration.

yarn.nodemanager.resource.percentage-physical-cpu-limit

100

Percentage of CPU that can be allocated for containers. This setting allows users to limit the amount of CPU that YARN containers use. Currently functional only on Linux using cgroups. The default is to use 100% of CPU.

[#11]

Hi @Ilia K,

You need to create the new config group under Yarn config and add the new nodemanagers into that group.

Then override the yarn nodemanger properties for cpu and memory.

[#12]

I know,

i just wanted to prevent managaing multuiple configurations