Support Questions

Find answers, ask questions, and share your expertise

cdh集群yarn可用内存180g,用户提交spark任务基于yarn提交只能使用120g

avatar
New Contributor

mrTao_1-1680859976042.pngmrTao_2-1680859991128.png

 

2 REPLIES 2

avatar
Community Manager

Welcome to the community @mrTao

According to google translate your issue is around "CDH cluster yarn can use 180g of available memory, and users can only use 120g to submit spark tasks based on yarn submission". The image also indicated that YARN is using 72 cores and 174GiB of memory. 

Perhaps @Bharati or @rrrajesh will have some input. 


Cy Jervis, Manager, Community Program
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Master Collaborator

Hi @mrTao 

 

It is not good idea to use entire memory Cluster YARN memory.  You can tune the memory from spark side by adjusting the memory using following parameters:

--conf spark.executor.instances=5
--conf spark.driver.memory=10g
--conf spark.driver.memoryOverhead=1g
--conf spark.executor.memory=10g
--conf spark.executor.memoryOverhead=1g

With the above memory configuration, YARN will allocate 66gb(Executor memory 11gb * Executor Instances(5) + Driver memory 11gb = 55g + 11g = 66g).

 

Better check your spark-submit once again and tune the above parameters according to your requirement.