question Re: What is the best practice for multiple python environments with Spark2 interpreter in Zeppelin in Support Questions

What is the best practice for multiple python environments with Spark2 interpreter in Zeppelin

LegallyBind — Thu, 23 Sep 2021 05:53:26 GMT

Hello

I was wondering how one would best implement the case where multiple users have their own python environment with their respective packages installed in a Spark2 interpreter. I have not implemented or tested the two approaches below yet but was wondering if I am on a completely wrong path and there would be a very easy and straightforward solution.

Two ways I could think of would be the following:

1.) For each notebook set the variable zeppelin.python

If one would instantiate the interpreter per user you could run the following at the start.

%spark2.conf zeppelin.python </your/users/python_env>

This would add some complexity for the users because they have to write down their environment paths but would otherwise be very easy to implement (If it really works without hiccups).

2.) Create a separate Interpreter for every user

This approach would be to create an interpreter for every user (%spark2 -> %spark2userName).

This would be relatively easy to explain to users and they don't need to remember any paths. They can support themselves since a universal "Just add your username behind %spark2" as instruction would suffice.

I have only dipped my toes slightly into creating new Interpreters so I have no clue how easy or complicated it would be to clone one.

Can you just copy one interpreter's fields and create a copy with new names? I could not find a lot of information on this so any pointers would help.

Of course, my approaches might be stupid so if you have an actual solution I am glad to read about it.

Re: What is the best practice for multiple python environments with Spark2 interpreter in Zeppelin

RangaReddy — Thu, 07 Oct 2021 04:34:41 GMT

Hi @LegallyBind

Please find the following tutorial.

https://community.cloudera.com/t5/Customer/How-to-use-multiple-versions-of-Python-in-Zeppelin/ta-p/271226

Re: What is the best practice for multiple python environments with Spark2 interpreter in Zeppelin

VidyaSargur — Mon, 11 Oct 2021 06:11:51 GMT

@LegallyBind, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Re: What is the best practice for multiple python environments with Spark2 interpreter in Zeppelin

LegallyBind — Mon, 11 Oct 2021 12:13:59 GMT

Thank you for reply @RangaReddy

The article describes how to create multiple interpreters for multiple versions of Python.

So this is the suggested best practice when one needs multiple Python environments for Spark Interpreters?

Re: What is the best practice for multiple python environments with Spark2 interpreter in Zeppelin

RangaReddy — Sat, 16 Oct 2021 06:55:46 GMT

@LegallyBind

For each python, you need to create separate interpreter.

Re: What is the best practice for multiple python environments with Spark2 interpreter in Zeppelin

VidyaSargur — Thu, 21 Oct 2021 17:36:17 GMT

@LegallyBind Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.