Support Questions
Find answers, ask questions, and share your expertise

Using PEX with pyspark

In our organisation we wanted to ship python dependencies to spark cluster using pex, can any one help me how to submit jobs to spark with pex files. I have tried the following command but it did not work, pyspark says no moudule when tried to import packages present in pex file.We are using spark2.4.0

cmd : ./spark-2.4.0-bin-hadoop2.7/bin/pyspark --conf spark.executorEnv.PEX_ROOT=./.pex --files dependecies.pex