Created 03-31-2023 08:41 AM
Hello,
trying to setup zeppelin + livy-on-spark3.
Livy-on-spark2 is up and r
spark3.sql runs fine, any spark3.pyspark code returns error "Interpreter died:".
What could be wrong?
Created on 09-05-2023 09:26 AM - edited 09-05-2023 06:48 PM
i have also encountered this weird situation interpreter died when using livy pyspark on spark3. I have lost for 2 days debugging this errors and i've found out that we need to:
1. In the "Spark 3" service on Cloudera manager portal, set spark pyspark python and python driver executable files configuration in the section "Spark 3 Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-defaults.conf" as below:
spark.pyspark.driver.python=/usr/bin/python3
spark.pyspark.python=/usr/bin/python3
2. Restart "livy for Spark 3" service on cloudera manager.
3. Restart the Zeppelin Livy Interpreter.
=> After being restarted, the zeppelin's livy interpreter on Spark 3 can execute %pyspark interpreter.
=> Note that, the python version to work with livy spark 3 on zeppelin must be less than or equal 3.7. otherwise it will generate error "required field "type_ignores" missing from Module" when executing pyspark script on Zeppelin.
Created 10-05-2023 01:44 AM
Hi Team,
Livy3 with Zeppelin Integration is not yet supported.
To use Spark3, you need to install python3 and needs to add the following parameters:
PYSPARK3_PYTHON
spark.yarn.appMasterEnv.PYSPARK3_PYTHON
Reference:
Created 06-02-2023 05:30 AM
I have exact same issue. any hints?
Created 06-02-2023 10:26 AM
@DarekLinek @Scout Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our Spark experts @Bharati and @Gopinath who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,Created on 09-05-2023 09:26 AM - edited 09-05-2023 06:48 PM
i have also encountered this weird situation interpreter died when using livy pyspark on spark3. I have lost for 2 days debugging this errors and i've found out that we need to:
1. In the "Spark 3" service on Cloudera manager portal, set spark pyspark python and python driver executable files configuration in the section "Spark 3 Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-defaults.conf" as below:
spark.pyspark.driver.python=/usr/bin/python3
spark.pyspark.python=/usr/bin/python3
2. Restart "livy for Spark 3" service on cloudera manager.
3. Restart the Zeppelin Livy Interpreter.
=> After being restarted, the zeppelin's livy interpreter on Spark 3 can execute %pyspark interpreter.
=> Note that, the python version to work with livy spark 3 on zeppelin must be less than or equal 3.7. otherwise it will generate error "required field "type_ignores" missing from Module" when executing pyspark script on Zeppelin.
Created 10-05-2023 01:44 AM
Hi Team,
Livy3 with Zeppelin Integration is not yet supported.
To use Spark3, you need to install python3 and needs to add the following parameters:
PYSPARK3_PYTHON
spark.yarn.appMasterEnv.PYSPARK3_PYTHON
Reference: