yarn and spark memory tuning for the jobs

[#1]

Hi Team,

Please suggest any guide or recommendations of tuning the memory parameters for spark jobs.

Eg: To process 500GB input data by spark job how much executors and memory required for each executor etc...

[#2]

A good starting point is to review mistake number 1, in the slideshare:
https://www.slideshare.net/SparkSummit/top-5-mistakes-when-writing-spark-applications-63071421
This gives a good starting point in tuning the cores, executor memory, etc.

[#3]

Hi @hanumanth 

You can also review this for spark tuning 

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/tuning-spark/topics/spark-admin_spark_tuning....

[#4]

@hanumanth Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks