How to pick number of executors , cores for each executor and executor memory

[#1]

Cluster Information:

10 Node cluster, each machine has 16 cores and 126.04 GB of RAM

My Question how to pick num-executors, executor-memory, executor-core, driver-memory, driver-cores

Job will run using Yarn as resource schdeuler

[#2 · Accepted solution]

Cloudera has a nice two part tuning guide. Attaching the links:

• http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/
• http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/

[#3]

Hi,

Whether those links that was provided helped to solve the issue

• http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/
• http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/

if it was fine can you please mark this forum as solved.

Thanks

AKR

[#4]

@Sivaprasanna - cloudera blog share above is not available, the link redirects to https://blog.cloudera.com/ any idea?

[#5]

@srinivas Nalanagula

https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/

https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-1/

[#6]

@srinivas Nalanagula

https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/

https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-1/