Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark 1.6.0 and 2.1 parallel. Reaching out for 2.1 with plain python code

Spark 1.6.0 and 2.1 parallel. Reaching out for 2.1 with plain python code

New Contributor

Environment: CDH 5.10, additional spark 2.1

 

Hello,

 

essentially my code looks like:

 

from pyspark import SparkConf

from pyspark import SparkContext

conf = SparkConf().setMaster('yarn').setAppName('test')

sc = SparkContext(conf=conf)

print sc

 

Running this skript from the shell with "python -m my.project.lib.script" will get me a spark 1.6.0 context. I´d like to get an 2.1 context at this place.

 

I can get a 2.1 context running the code with spark-submit: "spark2-submit /path/to/my/project/lib/script.py". Also i get a 2.1 context with "pyspark2". 

 

Any hints for me how to get a 2.1 context from a plain python script started from the shell with "python script.py"?

 

Thanks a lot

 

1 REPLY 1

Re: Spark 1.6.0 and 2.1 parallel. Reaching out for 2.1 with plain python code

Champion

I would dig into the spark2-submit and pyspark2 files.  They usually have code specific to making them work for the particular environment and service.  I would not be suprised if they simply set the env vars to the 2.1 library locations and maybe a few other changes.