Support Questions

Find answers, ask questions, and share your expertise

Can some one provide the exact details of spark confifuration

In the below spark configurations, I could see executor memory has declared as 16GB and the no of executors are 20.

is it like each executor will take 16 GB or the entire 20 executor will consume 16 GB

each executor means overall 320 GB for the spark program 20 *16 =320? kindly help on this

executor cores allocated as 8 . How the spark program will assume resources


Sample spark command:

SPARK_HOME/sbin
start-thriftserver.sh
--driver-memory 10g
--verbose
--master yarn
--executor-memory 16g
--num-executors 20
--executor-cores 8
--conf spark.hadoop.yarn.timeline-service.enabled=false
--conf spark.yarn.executor.memoryOverhead=5120
--conf spark.driver.maxResultSize=5g
--conf spark.sql.crossJoin.enabled=true
--conf spark.rpc.askTimeout=700
--conf spark.sql.broadcastTimeout=800
--conf spark.kryoserializer.buffer.max=768m

1 REPLY 1

Mentor

@sai venkatesh

Here is a good start, because of every cluster environment is different so one can't give you a configuration that will work optimally on your cluster.
Please read and understand the Distribution of Executors, Cores and Memory for a Spark Application