Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Need to understand impact for setting tez.task.scale.memory.enabled

Need to understand impact for setting tez.task.scale.memory.enabled

Expert Contributor

Hi,

I was going through the smart sense recommendation, which suggests to enable "tez.task.scale.memory.enabled".

I searched through Tez official documentation which says

Whether to scale down memory requested by each component if the total exceeds the available JVM memory

I am keen on understanding if we enable auto-scaling of memory for tasks, what are the disadvantages and possible advantages.

Thanks for sharing your experience.

Regards,

4 REPLIES 4

Re: Need to understand impact for setting tez.task.scale.memory.enabled

Contributor

@Smart Solutions

Were you able to get an answer for this?

Re: Need to understand impact for setting tez.task.scale.memory.enabled

Expert Contributor

Re: Need to understand impact for setting tez.task.scale.memory.enabled

Rising Star

What I understand from documentation (I am applying same changes) and from previous experience is that happens that some jobs has too many inputs and outputs to be handle by the system itself (mappers, containers, reducers etc...) this affects from one side your cluster performance since resources are not enough for such jobs to complete or simply they start to present a considerable delay to what is expected. I've seen that's the moment when OOM error shows up.

Saying this, I guess explanation from the smartsense is clear since it tells us "Enabling auto-scaling scales down requests, which may otherwise be too large."

Hope this helps ;)