we have multiple systems with different cpu and ram each
the cpu can be used to utilize 100%
but i could not do it for the ram
You can manually configure your memory allocation settings based on the hardware specifications. Please refer below links for more tuning up RAM in yarn configuration.
For changing memory value through ambari configuration manager, login into ambari UI,
Yarn —> Configs —> Settings —> memory —> Memory allocated for all YARN containers on a node and Minimum / Maximum Container Size memory.
and for advanced turning of yarn modify
Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site
If you set 100% ram only for yarn then other services will get crash / GC issue. It's not recommendable to set all physical memory to one component.
For example your physical memory size is 10GB you need to allocated for every container request at the RM, is around 9GB (90% of your physical memory) then you can mention in below parameter
Similarly amount of physical memory allocated for containers in node manager, that can be mention using below parameter.
These all settings are manage by single configuration file I.e., yarn-site.xml which you can modify via ambari under Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site
I hope that the above answers your questions. Please accept the answer you found most useful.
i know to to configure it,
for example now i have 3 types of hardares:
3 computers with 12 core cpu and 64gb ram
6 computers with 16 core and 64gb ram
and 4 computers with 48 core and 256gb ram
and i dont want to manage 3 configuraion files , i want to manage one, for instance each computer will allocate all cpus and 90% of ram for yarn tasks
For your question, answer is No. As of now we can't mange memory of individual node level using percentage. Currently yarn support only CPU, using below configuration.
|yarn.nodemanager.resource.percentage-physical-cpu-limit||100||Percentage of CPU that can be allocated for containers. This setting allows users to limit the amount of CPU that YARN containers use. Currently functional only on Linux using cgroups. The default is to use 100% of CPU.|