question Re: livy-on-spark3 interpreter died in Support Questions

livy-on-spark3 interpreter died

Scout — Tue, 21 Apr 2026 07:16:44 GMT

Hello,

trying to setup zeppelin + livy-on-spark3.

Livy-on-spark2 is up and r

spark3.sql runs fine, any spark3.pyspark code returns error "Interpreter died:".

What could be wrong?

Re: livy-on-spark3 interpreter died

DarekLinek — Fri, 02 Jun 2023 12:30:39 GMT

I have exact same issue. any hints?

Re: livy-on-spark3 interpreter died

DianaTorres — Fri, 02 Jun 2023 17:26:18 GMT

@DarekLinek @Scout Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Spark experts @Bharati and @Gopinath who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: livy-on-spark3 interpreter died

Hai1 — Wed, 06 Sep 2023 01:48:37 GMT

i have also encountered this weird situation interpreter died when using livy pyspark on spark3. I have lost for 2 days debugging this errors and i've found out that we need to:

1. In the "Spark 3" service on Cloudera manager portal, set spark pyspark python and python driver executable files configuration in the section "Spark 3 Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-defaults.conf" as below:

spark.pyspark.driver.python=/usr/bin/python3
spark.pyspark.python=/usr/bin/python3

2. Restart "livy for Spark 3" service on cloudera manager.

3. Restart the Zeppelin Livy Interpreter.

=> After being restarted, the zeppelin's livy interpreter on Spark 3 can execute %pyspark interpreter.

=> Note that, the python version to work with livy spark 3 on zeppelin must be less than or equal 3.7. otherwise it will generate error "required field "type_ignores" missing from Module" when executing pyspark script on Zeppelin.

Re: livy-on-spark3 interpreter died

RangaReddy — Thu, 05 Oct 2023 08:44:45 GMT

Hi Team,

Livy3 with Zeppelin Integration is not yet supported.

To use Spark3, you need to install python3 and needs to add the following parameters:

PYSPARK3_PYTHON

spark.yarn.appMasterEnv.PYSPARK3_PYTHON

Reference:

https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/running-spark-applications/topics/spark-python-path-variables-livy.html