In the below spark configurations, I could see executor memory has declared as 16GB and the no of executors are 20.
is it like each executor will take 16 GB or the entire 20 executor will consume 16 GB
each executor means overall 320 GB for the spark program 20 *16 =320? kindly help on this
executor cores allocated as 8 . How the spark program will assume resources
Sample spark command:
SPARK_HOME/sbinstart-thriftserver.sh--driver-memory 10g--verbose--master yarn--executor-memory 16g--num-executors 20--executor-cores 8--conf spark.hadoop.yarn.timeline-service.enabled=false--conf spark.yarn.executor.memoryOverhead=5120--conf spark.driver.maxResultSize=5g--conf spark.sql.crossJoin.enabled=true--conf spark.rpc.askTimeout=700--conf spark.sql.broadcastTimeout=800--conf spark.kryoserializer.buffer.max=768m
Here is a good start, because of every cluster environment is different so one can't give you a configuration that will work optimally on your cluster.Please read and understand the Distribution of Executors, Cores and Memory for a Spark Application