06-22-2017 02:14 AM
Environment: CDH 5.10, additional spark 2.1
essentially my code looks like:
from pyspark import SparkConf
from pyspark import SparkContext
conf = SparkConf().setMaster('yarn').setAppName('test')
sc = SparkContext(conf=conf)
Running this skript from the shell with "python -m my.project.lib.script" will get me a spark 1.6.0 context. I´d like to get an 2.1 context at this place.
I can get a 2.1 context running the code with spark-submit: "spark2-submit /path/to/my/project/lib/script.py". Also i get a 2.1 context with "pyspark2".
Any hints for me how to get a 2.1 context from a plain python script started from the shell with "python script.py"?
Thanks a lot
06-23-2017 06:25 AM
I would dig into the spark2-submit and pyspark2 files. They usually have code specific to making them work for the particular environment and service. I would not be suprised if they simply set the env vars to the 2.1 library locations and maybe a few other changes.