Member since
08-04-2017
6
Posts
0
Kudos Received
0
Solutions
05-08-2019
04:34 AM
@jtaras, thank you for this wonderful article! I have a follow-up question (just curious) : do Spark workers use the same custom Python path as Zeppelin driver? If yes, how do they know it? (the Zeppelin UI setting seems to apply to the Zeppelin driver application only and its SparkContext) If no, why is there no version conflict between the driver's python and the workers' python? (I tested with the default of 2.X and the custom Anaconda python of 3.7) NOTE: I had to additionally set the PYSPARK_PYTHON environment variable in Spark > Config to the same path in Ambari, in order to be able to use all the Python libraries in the scripts submitted via "spark-submit", but this does not affect the Zeppelin functionality in any way.
... View more