Reply
New Contributor
Posts: 1
Registered: ‎06-22-2017

Spark 1.6.0 and 2.1 parallel. Reaching out for 2.1 with plain python code

Environment: CDH 5.10, additional spark 2.1

 

Hello,

 

essentially my code looks like:

 

from pyspark import SparkConf

from pyspark import SparkContext

conf = SparkConf().setMaster('yarn').setAppName('test')

sc = SparkContext(conf=conf)

print sc

 

Running this skript from the shell with "python -m my.project.lib.script" will get me a spark 1.6.0 context. I´d like to get an 2.1 context at this place.

 

I can get a 2.1 context running the code with spark-submit: "spark2-submit /path/to/my/project/lib/script.py". Also i get a 2.1 context with "pyspark2". 

 

Any hints for me how to get a 2.1 context from a plain python script started from the shell with "python script.py"?

 

Thanks a lot

 

Highlighted
Posts: 642
Topics: 3
Kudos: 105
Solutions: 67
Registered: ‎08-16-2016

Re: Spark 1.6.0 and 2.1 parallel. Reaching out for 2.1 with plain python code

I would dig into the spark2-submit and pyspark2 files.  They usually have code specific to making them work for the particular environment and service.  I would not be suprised if they simply set the env vars to the 2.1 library locations and maybe a few other changes.

Announcements