question Import error on pyspark and Zeppelin for local module in Archives of Support Questions (Read Only)

Import error on pyspark and Zeppelin for local module

bharatbs13 — Fri, 25 May 2018 18:00:50 GMT

I am moving to pyspark & zeppelin . I created a two notebook my_settings.py and main.py .

But when I do following in main.py

%pyspark 
import my_settings 

ImportError: No module named my_settings

I get import error saying , no modules found.

This works fine on my local server.

I wonder if there's any env setting for this to work.

Re: Import error on pyspark and Zeppelin for local module

falbani — Fri, 25 May 2018 22:21:42 GMT

@bharat sharma Notebooks are not python modules. If you are trying to import a notebook as if it was a python module AFAIK that won't work.

If you are trying to import modules to pyspark application you have different ways to do this. One way is to copy the python file to hdfs and use the following:

%pyspark
sc.addPyFile("/user/zeppelin/my_settings.py")
import my_settings

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: Import error on pyspark and Zeppelin for local module

falbani — Mon, 28 May 2018 21:18:01 GMT

@bharat sharma If the above answer helped addressed your question, please take a moment to login and click the "accept" link on the answer.