Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can some one provide the exact details of spark confifuration

Highlighted

Can some one provide the exact details of spark confifuration

New Contributor

In the below spark configurations, I could see executor memory has declared as 16GB and the no of executors are 20.

is it like each executor will take 16 GB or the entire 20 executor will consume 16 GB

each executor means overall 320 GB for the spark program 20 *16 =320? kindly help on this

executor cores allocated as 8 . How the spark program will assume resources


Sample spark command:

SPARK_HOME/sbin
start-thriftserver.sh
--driver-memory 10g
--verbose
--master yarn
--executor-memory 16g
--num-executors 20
--executor-cores 8
--conf spark.hadoop.yarn.timeline-service.enabled=false
--conf spark.yarn.executor.memoryOverhead=5120
--conf spark.driver.maxResultSize=5g
--conf spark.sql.crossJoin.enabled=true
--conf spark.rpc.askTimeout=700
--conf spark.sql.broadcastTimeout=800
--conf spark.kryoserializer.buffer.max=768m

1 REPLY 1

Re: Can some one provide the exact details of spark confifuration

Mentor

@sai venkatesh

Here is a good start, because of every cluster environment is different so one can't give you a configuration that will work optimally on your cluster.
Please read and understand the Distribution of Executors, Cores and Memory for a Spark Application