We are trying to use graphframes package with pyspark. For some reason it doesn't work in our production environment. In our dev environment it works as we can use --packages options and it downloads the libraries from external repository. We cannot use packages option in production as it is not connected to the internet. It works with scala in production. The default python version is 2.6.6 and hdp version is 2.4.2 pyspark --packages graphframes:graphframes:0.2.0-spark1.6-s_2.10 I copied the all the jars downloaded with --packages option in dev and passed it as parameter to --jars in pyspark command in production. But it doesn't work. The same commands work in dev and spark on my mac. pyspark --py-files /tmp/thirdpartyjars/graphframes_graphframes-0.2.0-spark1.6-s_2.10.jar,/tmp/thirdpartyjars/com.typesafe.scala-logging_scala-logging-api_2.10-2.1.2.jar,/tmp/thirdpartyjars/com.typesafe.scala-logging_scala-logging-slf4j_2.10-2.1.2.jar,/tmp/thirdpartyjars/org.scala-lang_scala-reflect-2.10.4.jar,/tmp/thirdpartyjars/org.slf4j_slf4j-api-1.7.7.jar --jars /tmp/thirdpartyjars/graphframes_graphframes-0.2.0-spark1.6-s_2.10.jar,/tmp/thirdpartyjars/com.typesafe.scala-logging_scala-logging-api_2.10-2.1.2.jar,/tmp/thirdpartyjars/com.typesafe.scala-logging_scala-logging-api_2.10-2.1.2.jar,/tmp/thirdpartyjars/com.typesafe.scala-logging_scala-logging-slf4j_2.10-2.1.2.jar,/tmp/thirdpartyjars/org.scala-lang_scala-reflect-2.10.4.jar,/tmp/thirdpartyjars/org.slf4j_slf4j-api-1.7.7.jar Console log
Using Python version 2.6.6 (r266:84292, May 22 2015 08:34:51)
SparkContext available as sc, HiveContext available as sqlContext.
>>>
>>> from graphframes import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
zipimport.ZipImportError: can't find module 'graphframes'
... View more
