question Re: Spark 2 interpreter runs only 3 containers in Archives of Support Questions (Read Only)

Spark 2 interpreter runs only 3 containers

grabowski14 — Sun, 18 Aug 2019 08:59:12 GMT

Hi. I have a problem with Spark 2 interpreter in Zeppelin. I configured interpreter like this:

When I run query like this:

%spark2.sql
select var1, count(*) as counter
from database.table_1
group by var1
order by counter desc

Spark job runs only 3 containers and job takes 13 minutes.

Does anyone know why Spark interpreter takes only 4.9 % of queue? How I should configure the interpreter to increase this factor?

Re: Spark 2 interpreter runs only 3 containers

yvora — Tue, 18 Jul 2017 04:24:34 GMT

@Mateusz Grabowski, You should enable Dynamic Resource Allocation in Spark to automatically increase/decrease executors of an app as per resource availability.

You can choose to enable DRA in either Spark or Zeppelin .

1) Enable DRA for Spark2 as below.

https://community.hortonworks.com/content/supportkb/49510/how-to-enable-dynamic-resource-allocation-in-spark.html

2) Enable DRA via Livy Interpreter. Run all spark notebooks via livy interpreters.

https://zeppelin.apache.org/docs/0.6.1/interpreter/livy.html

Re: Spark 2 interpreter runs only 3 containers

grabowski14 — Tue, 18 Jul 2017 17:01:23 GMT

It works! Thank you 🙂