Hi @hkropp I followed your post and when I trigger my spark-submit and the application is accepted by the yarn cluster I get the following error: Do you know what might be the reason? In STDOUT ./CONDA_TEST/test_env3/bin/python: ./CONDA_TEST/test_env3/bin/python: cannot execute binary file In STDERR 19/02/11 10:48:43 INFO ApplicationMaster: Preparing Local resources 19/02/11 10:48:44 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1548016788605_1234_000002 19/02/11 10:48:44 INFO SecurityManager: Changing view acls to: mbilling 19/02/11 10:48:44 INFO SecurityManager: Changing modify acls to: mbilling 19/02/11 10:48:44 INFO SecurityManager: Changing view acls groups to: 19/02/11 10:48:44 INFO SecurityManager: Changing modify acls groups to: 19/02/11 10:48:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mbilling); groups with view permissions: Set(); users with modify permissions: Set(mbilling); groups with modify permissions: Set() 19/02/11 10:48:44 INFO ApplicationMaster: Starting the user application in a separate Thread 19/02/11 10:48:44 INFO ApplicationMaster: Waiting for spark context initialization... 19/02/11 10:48:44 ERROR ApplicationMaster: User application exited with status 126 19/02/11 10:48:44 INFO ApplicationMaster: Final app status: FAILED, exitCode: 126, (reason: User application exited with status 126) 19/02/11 10:48:44 ERROR ApplicationMaster: Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:194) at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:428) at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:281) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:783) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:781) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) Caused by: org.apache.spark.SparkUserAppException: User application exited with 126 at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:104) at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:654) 19/02/11 10:48:44 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User application exited with status 126) 19/02/11 10:48:44 INFO ApplicationMaster: Deleting staging directory hdfs://pmids01/user/mbilling/.sparkStaging/application_1548016788605_1234 19/02/11 10:48:44 INFO ShutdownHookManager: Shutdown hook called Im running like this export SPARK_MAJOR_VERSION=2; \ export PYSPARK_PYTHON=./CONDA_TEST/test_env3/bin/python; \ export PYSPARK_DRIVER_PYTHON=./CONDA_TEST/test_env3/bin/python; \ export PYSPARK_DRIVER_PYTHON_OPTS=''; \ cd deploy;\ spark-submit --master yarn --deploy-mode cluster \ --verbose \ --num-executors 1 --driver-memory 1g --executor-memory 1g \ --files /usr/hdp/current/spark2-client/conf/hive-site.xml \ --jars /usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar \ --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./CONDA_TEST/test_env3/bin/python \ --archives test_env3.zip#CONDA_TEST \ --py-files main.py main.py ... View more