Support Questions
Find answers, ask questions, and share your expertise

Conda virtual env and spark-submit ?

Conda virtual env and spark-submit ?

Explorer

Hello,

 

I'm running Spark 2.3.0 installed from the parcel and activated on a cluster.

After creating conda virtual env, and activating it, I've noticed that spark-submit is called from /opt/cloudera/parcels/Anaconda/envs/py37/bin/spark-submit and the "spark-submit --version" returns "version 2.4.0."

Could someone explain a little bit - does it mean that spark is installed with Anaconda/pyspark ? Because version 2.4.0 can't be seen anywhere on the system.

 

Regards,

Nenad