This line in the console output indicates the job was rejected by YARN: 18/11/12 11:20:25 ERROR spark.SparkContext: Error initializing SparkContext. org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1537380011359_1185346 to YARN : Application rejected by queue placement policy   YARN determines which resource pool to place jobs in via placement rules in the scheduler configuration (see https://blog.cloudera.com/blog/2016/06/untangling-apache-hadoop-yarn-part-4-fair-scheduler-queue-basics/).   It might be worth checking whether you have a "<rule name="reject"/>" rule in your scheduler configuration and if so, why the user's pyspark job reached this rule instead of one of the previous rules to place the job in the appropriate queue. ... View more