Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

add zip folder and import specific module / class

add zip folder and import specific module / class

New Contributor

I have a project structure like

foo/foo/apps/lib/math.py

and there are other .py files .

When I do

sc.addPyFile("foo.zip")

How Can I import math.py or any class of of this.

3 REPLIES 3

Re: add zip folder and import specific module / class

Super Collaborator

I think you will have to add the complete path, not only the file name:

addPyFile(path)[source]

Add a .py or .zip dependency for all tasks to be executed on this SparkContext in the future. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI.

Re: add zip folder and import specific module / class

New Contributor

yes I can add zip file along with path. But there are modules inside compressed folder. How do i Import this.

On python I can do . I have __init__.py in all folders starting from second 'foo'

import foo.apps.lib.math

Re: add zip folder and import specific module / class

Super Collaborator

in pyspark addPyfile should be working in the same way as in python directly. So maybe you can provide more details on your issue? The addPyfile is wokring? But the import fails? Do you get an error message?