Need to understand impact for setting tez.task.scale.memory.enabled

[#1]

Hi,

I was going through the smart sense recommendation, which suggests to enable "tez.task.scale.memory.enabled".

I searched through Tez official documentation which says

Whether to scale down memory requested by each component if the total exceeds the available JVM memory

I am keen on understanding if we enable auto-scaling of memory for tasks, what are the disadvantages and possible advantages.

Thanks for sharing your experience.

Regards,

[#2]

@Smart Solutions

Were you able to get an answer for this?

[#3]

Nope @Josh Nicholson

[#4]

What I understand from documentation (I am applying same changes) and from previous experience is that happens that some jobs has too many inputs and outputs to be handle by the system itself (mappers, containers, reducers etc...) this affects from one side your cluster performance since resources are not enough for such jobs to complete or simply they start to present a considerable delay to what is expected. I've seen that's the moment when OOM error shows up.

Saying this, I guess explanation from the smartsense is clear since it tells us "Enabling auto-scaling scales down requests, which may otherwise be too large."

Hope this helps 😉

[#5]

Also check

https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html

https://community.hortonworks.com/questions/5780/hive-on-tez-query-map-output-outofmemoryerror-java....

https://community.hortonworks.com/questions/12067/what-is-the-workaround-when-getting-hive-outofmemo...