Support Questions
Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Need to understand impact for setting tez.task.scale.memory.enabled

Expert Contributor


I was going through the smart sense recommendation, which suggests to enable "tez.task.scale.memory.enabled".

I searched through Tez official documentation which says

Whether to scale down memory requested by each component if the total exceeds the available JVM memory

I am keen on understanding if we enable auto-scaling of memory for tasks, what are the disadvantages and possible advantages.

Thanks for sharing your experience.




@Smart Solutions

Were you able to get an answer for this?

Expert Contributor

Rising Star

What I understand from documentation (I am applying same changes) and from previous experience is that happens that some jobs has too many inputs and outputs to be handle by the system itself (mappers, containers, reducers etc...) this affects from one side your cluster performance since resources are not enough for such jobs to complete or simply they start to present a considerable delay to what is expected. I've seen that's the moment when OOM error shows up.

Saying this, I guess explanation from the smartsense is clear since it tells us "Enabling auto-scaling scales down requests, which may otherwise be too large."

Hope this helps 😉