Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Need to understand impact for setting tez.task.scale.memory.enabled

avatar
Expert Contributor

Hi,

I was going through the smart sense recommendation, which suggests to enable "tez.task.scale.memory.enabled".

I searched through Tez official documentation which says

Whether to scale down memory requested by each component if the total exceeds the available JVM memory

I am keen on understanding if we enable auto-scaling of memory for tasks, what are the disadvantages and possible advantages.

Thanks for sharing your experience.

Regards,

4 REPLIES 4

avatar
Expert Contributor

@Smart Solutions

Were you able to get an answer for this?

avatar
Expert Contributor

avatar
Expert Contributor

What I understand from documentation (I am applying same changes) and from previous experience is that happens that some jobs has too many inputs and outputs to be handle by the system itself (mappers, containers, reducers etc...) this affects from one side your cluster performance since resources are not enough for such jobs to complete or simply they start to present a considerable delay to what is expected. I've seen that's the moment when OOM error shows up.

Saying this, I guess explanation from the smartsense is clear since it tells us "Enabling auto-scaling scales down requests, which may otherwise be too large."

Hope this helps 😉