Hi. I have a problem with Spark 2 interpreter in Zeppelin. I configured interpreter like this:
When I run query like this:
select var1, count(*) as counter
group by var1
order by counter desc
Spark job runs only 3 containers and job takes 13 minutes.
Does anyone know why Spark interpreter takes only 4.9 % of queue? How I should configure the interpreter to increase this factor?
@Mateusz Grabowski, You should enable Dynamic Resource Allocation in Spark to automatically increase/decrease executors of an app as per resource availability.
You can choose to enable DRA in either Spark or Zeppelin .
1) Enable DRA for Spark2 as below.
2) Enable DRA via Livy Interpreter. Run all spark notebooks via livy interpreters.
View solution in original post
It works! Thank you 🙂