question Re: What are Spark executors, executor instances, executor_cores, worker threads, worker nodes and number of executors? in Archives of Support Questions (Read Only)

What are Spark executors, executor instances, executor_cores, worker threads, worker nodes and number of executors?

sree_kuppa — Tue, 04 Apr 2017 04:22:26 GMT

Additionally, what exactly does dynamic allocation mean?? (I know it means allocating containers/executors on the fly but please elaborate) What are "spark.dynamicAllocation.maxExecutors"?? What should its value be?

How are each of these parameters related to each other?? What should be the setting of all these parameters for best performance on the cluster?

I read the documentation but I am afraid I am still confused with the terminology. Really appreciate if someone can elaborate.

Re: What are Spark executors, executor instances, executor_cores, worker threads, worker nodes and number of executors?

kbadani — Wed, 05 Apr 2017 00:54:58 GMT

@Sree Kupp

This is a great video , explaining everything in detail. Its a 6 hour training but you can skip and listen to the parts that you are interested in.

https://www.youtube.com/watch?v=7ooZ4S7Ay6Y

Re: What are Spark executors, executor instances, executor_cores, worker threads, worker nodes and number of executors?

sree_kuppa — Wed, 05 Apr 2017 02:47:17 GMT

Thanks @Kshitij Badani.. will look into it.

Re: What are Spark executors, executor instances, executor_cores, worker threads, worker nodes and number of executors?

kbadani — Wed, 05 Apr 2017 05:49:37 GMT

@Sree Kupp The part you are interested in starts after 1:20:00, and upto 2:40:00

Re: What are Spark executors, executor instances, executor_cores, worker threads, worker nodes and number of executors?

sree_kuppa — Tue, 11 Apr 2017 00:53:25 GMT

That was a great video @Kshitij Badani. Thanks for the pointers.