Spark 2 interpreter runs only 3 containers

[#1]

Hi. I have a problem with Spark 2 interpreter in Zeppelin. I configured interpreter like this:

When I run query like this:

%spark2.sql
select var1, count(*) as counter
from database.table_1
group by var1
order by counter desc

Spark job runs only 3 containers and job takes 13 minutes.

Does anyone know why Spark interpreter takes only 4.9 % of queue? How I should configure the interpreter to increase this factor?

[#2 · Accepted solution]

@Mateusz Grabowski, You should enable Dynamic Resource Allocation in Spark to automatically increase/decrease executors of an app as per resource availability.

You can choose to enable DRA in either Spark or Zeppelin .

1) Enable DRA for Spark2 as below.

https://community.hortonworks.com/content/supportkb/49510/how-to-enable-dynamic-resource-allocation-...

2) Enable DRA via Livy Interpreter. Run all spark notebooks via livy interpreters.

https://zeppelin.apache.org/docs/0.6.1/interpreter/livy.html

[#3]

It works! Thank you 🙂