question Re: YARN Tuning - How is memory overhead estimate determined? in Archives of Support Questions (Read Only)

YARN Tuning - How is memory overhead estimate determined?

MilesYao — Fri, 16 Sep 2022 09:35:51 GMT

Looking at the "Tuning YARN" documentation page, I noticed that the estimate for "Task Overhead" seems very large - 51GB for the example given, and 24GB on the tuning spreadsheet. The s/s comment says: "Allow additional overhead for task buffers, such as the HDFS Sort I/O buffer, JVM overheads etc." However, it does not explain how the number is derived. This seems steep for a small/medium cluster that probably account for the majority of users. Similar config guide at Hortonworks uses a rough ratio of 1/8 instead.

Please help - I need justification for requesting additional hardware $$s.

Thanks,

Miles

Re: YARN Tuning - How is memory overhead estimate determined?

Harsh J — Fri, 18 Sep 2015 11:59:15 GMT

The value on the doc page is picked as about 20% of the RAM for overhead reservation, but you could set it lower. Our past overcommit testing does show that the values can reach close to extra 20% in use for some tested workloads, but that would not be an always-as-such case - and this may have changed overall lately also.

We're reworking the docs for these recommendations soon in future, as developments happen. For now, please rely on the XLSX file for a more closer guideline on the recommended calculated values.