question Re: PySpark in Zeppelin: Does not have all libraries in Archives of Support Questions (Read Only)

PySpark in Zeppelin: Does not have all libraries

vjain — Wed, 09 Dec 2015 22:46:48 GMT

Iam able to import a library in pyspark shell without any problems, but when I try to import the same library in Zeppelin, I get an error

ImportError: No module named xxxxx

Re: PySpark in Zeppelin: Does not have all libraries

abajwa — Wed, 09 Dec 2015 22:52:00 GMT

Do you have multiple Python versions installed?

Re: PySpark in Zeppelin: Does not have all libraries

nsabharwal — Wed, 09 Dec 2015 23:05:05 GMT

@Vedant Jain

Please check this http://stackoverflow.com/questions/13212987/cannot...

Re: PySpark in Zeppelin: Does not have all libraries

vjain — Wed, 09 Dec 2015 23:20:06 GMT

I had 2 versions of Python installed. Zeppelin is still using the older one.

Re: PySpark in Zeppelin: Does not have all libraries

abajwa — Thu, 10 Dec 2015 00:58:13 GMT

For anyone who else may encounter this issue and end up here: this is most commonly the result of having multiple python versions installed. However if you are using Zeppelin (which is the case here), it is pretty easy to point to different version of python. In Zeppelin UI > Interpreter > Spark > Change the 'zeppelin.pyspark.python' property from 'python' to '/path/to/correct/pythondir/python' and click Save

Re: PySpark in Zeppelin: Does not have all libraries

vjain — Thu, 10 Dec 2015 03:55:41 GMT

@Ali Bajwa

Should it be the python directory or the pyspark directory?

as in /usr/loca/../python or /usr/hdp/2..../spark/python

Re: PySpark in Zeppelin: Does not have all libraries

abajwa — Thu, 10 Dec 2015 13:13:47 GMT

@Vedant Jain: I believe it should be the full path to the python executable you wish to use (assuming you don't want to use the default)