Support Questions
Find answers, ask questions, and share your expertise

How to find number of containers in context of spark and yarn? How exactly memory per container is calculated? How to solve container memory beyond physical limits?

Highlighted

How to find number of containers in context of spark and yarn? How exactly memory per container is calculated? How to solve container memory beyond physical limits?

Explorer
 
2 REPLIES 2
Highlighted

Re: How to find number of containers in context of spark and yarn? How exactly memory per container is calculated? How to solve container memory beyond physical limits?

Expert Contributor

@Dhana Shekar there is a description at https://community.hortonworks.com/questions/29469/yarn-container-size-flexible-to-satisfy-what-appli... that might help with a couple of your questions.

Highlighted

Re: How to find number of containers in context of spark and yarn? How exactly memory per container is calculated? How to solve container memory beyond physical limits?

Explorer

Thanks much for timely reply. Very useful. Just found below links also useful.

https://community.hortonworks.com/articles/42803/spark-on-yarn-executor-resource-allocation-optimiz....