Reply
Highlighted
Explorer
Posts: 14
Registered: ‎06-27-2017

Hive on Spark with Spark Dynamic Resource Allocation

In Hive configuration we have 
spark.dynamicAllocation.enabled = true, actually checked

spark.executor.cores = 6

Does dynamic override the set number of 6? It does not appear dynamic cores are being allocated.

 

spark_cores.png

Cloudera Employee
Posts: 122
Registered: ‎03-01-2016

Re: Hive on Spark with Spark Dynamic Resource Allocation

Hi

 

Spark dynamic allocation allows spark to add and remove the executors dynamically based on your application's demand. But for each executor, it will still respect your settings (e.g. cores and RAM per executor) and use a unified and fixed spec. 

 

Hope it helps.

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: Hive on Spark with Spark Dynamic Resource Allocation

Dynamic Allocation [1] controls the number of parallel running executor
containers in YARN, not the number of CPU vcore resource allocated to a
single executor container.

[1] -
https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.