question Re: yarn configure to use all ram in Support Questions

yarn configure to use all ram

ilia987 — Tue, 25 Dec 2018 18:02:29 GMT

we have multiple systems with different cpu and ram each

the cpu can be used to utilize 100%

but i could not do it for the ram

HDP-2.6.4.0

Re: yarn configure to use all ram

jagadeesan — Wed, 26 Dec 2018 14:45:39 GMT

@Ilia K

You can manually configure your memory allocation settings based on the hardware specifications. Please refer below links for more tuning up RAM in yarn configuration.

https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_command-line-installation/content/determine-hdp-memory-config.htm

Re: yarn configure to use all ram

ilia987 — Wed, 26 Dec 2018 15:44:35 GMT

i want to configure it via the ambari configuraion manager, so theat each host will use 100% ram and CPU

Re: yarn configure to use all ram

jagadeesan — Wed, 26 Dec 2018 16:37:01 GMT

For changing memory value through ambari configuration manager, login into ambari UI,

Yarn —> Configs —> Settings —> memory —> Memory allocated for all YARN containers on a node and Minimum / Maximum Container Size memory.

and for advanced turning of yarn modify

Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site

Re: yarn configure to use all ram

ilia987 — Wed, 26 Dec 2018 17:19:20 GMT

i looking for an option to set 100% and not to set fixd ram or cpu

Re: yarn configure to use all ram

jagadeesan — Wed, 26 Dec 2018 18:05:02 GMT

If you set 100% ram only for yarn then other services will get crash / GC issue. It's not recommendable to set all physical memory to one component.

Re: yarn configure to use all ram

ilia987 — Wed, 26 Dec 2018 18:23:35 GMT

ok, then configure a 90% mark, the main issue that i want to manage single configuraion file for all hardware

Re: yarn configure to use all ram

jagadeesan — Wed, 26 Dec 2018 18:48:16 GMT

For example your physical memory size is 10GB you need to allocated for every container request at the RM, is around 9GB (90% of your physical memory) then you can mention in below parameter

yarn.scheduler.maximum-allocation-mb 9216

Similarly amount of physical memory allocated for containers in node manager, that can be mention using below parameter.

yarn.nodemanager.resource.memory-mb 9216

These all settings are manage by single configuration file I.e., yarn-site.xml which you can modify via ambari under Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site

I hope that the above answers your questions. Please accept the answer you found most useful.

Re: yarn configure to use all ram

ilia987 — Wed, 26 Dec 2018 18:54:31 GMT

i know to to configure it,

for example now i have 3 types of hardares:

3 computers with 12 core cpu and 64gb ram

6 computers with 16 core and 64gb ram

and 4 computers with 48 core and 256gb ram

and i dont want to manage 3 configuraion files , i want to manage one, for instance each computer will allocate all cpus and 90% of ram for yarn tasks

Re: yarn configure to use all ram

jagadeesan — Wed, 26 Dec 2018 19:20:40 GMT

For your question, answer is No. As of now we can't mange memory of individual node level using percentage. Currently yarn support only CPU, using below configuration.

yarn.nodemanager.resource.percentage-physical-cpu-limit

100

Percentage of CPU that can be allocated for containers. This setting allows users to limit the amount of CPU that YARN containers use. Currently functional only on Linux using cgroups. The default is to use 100% of CPU.

Re: yarn configure to use all ram

subhash_parise3 — Thu, 27 Dec 2018 15:59:49 GMT

Hi @Ilia K,

You need to create the new config group under Yarn config and add the new nodemanagers into that group.

Then override the yarn nodemanger properties for cpu and memory.

Re: yarn configure to use all ram

ilia987 — Thu, 27 Dec 2018 18:49:19 GMT

I know,

i just wanted to prevent managaing multuiple configurations