I'm running spark-submit using the following command: PYSPARK_PYTHON=./ROOT/myspark/bin/python /usr/hdp/current/spark2-client/bin/spark-submit \ --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./ROOT/myspark/bin/python \ --master=yarn \ --deploy-mode=cluster \ --driver-memory=4g \ --archives=myspark.zip#ROOT \ --num-executors=32 \ --packages com.databricks:spark-avro_2.11:4.0.0 \ foo.py myspark.zip is a zipped conda environment. It was created using python with the zipfile pacakge. The files are stored without deflation. foo.py is my application code. This normally works, but if myspark.zip is greater than 2Gb I get: java.util.zip.ZipException: invalid CEN header (bad signature) My java version is: jdk1.8.0_112 It looks like older versions of java had this issue, but not my current one. I've written a test java class using java.util.zip which is able to unzip myspark.zip ok. I've checked that all my processes use the above version of java. YARN logs on console after above command. I've tried both --deploy-mode=cluster and --deploy-mode=client: 18/06/13 16:00:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/06/13 16:00:23 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 18/06/13 16:00:23 INFO RMProxy: Connecting to ResourceManager at myhost2.myfirm.com/10.87.11.17:8050 18/06/13 16:00:23 INFO Client: Requesting a new application from cluster with 6 NodeManagers 18/06/13 16:00:23 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (221184 MB per container) 18/06/13 16:00:23 INFO Client: Will allocate AM container, with 18022 MB memory including 1638 MB overhead 18/06/13 16:00:23 INFO Client: Setting up container launch context for our AM 18/06/13 16:00:23 INFO Client: Setting up the launch environment for our AM container 18/06/13 16:00:23 INFO Client: Preparing resources for our AM container 18/06/13 16:00:24 INFO Client: Use hdfs cache file as spark.yarn.archive for HDP, hdfsCacheFile:hdfs://myhost.myfirm.com:8020/hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz 18/06/13 16:00:24 INFO Client: Source and destination file systems are the same. Not copying hdfs://myhost.myfirm.com:8020/hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz 18/06/13 16:00:24 INFO Client: Uploading resource file:/home/myuser/.ivy2/jars/com.databricks_spark-avro_2.11-4.0.0.jar -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/com.databri cks_spark-avro_2.11-4.0.0.jar 18/06/13 16:00:26 INFO Client: Uploading resource file:/home/myuser/.ivy2/jars/org.slf4j_slf4j-api-1.7.5.jar -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.slf4j_slf4j-api-1. 7.5.jar 18/06/13 16:00:26 INFO Client: Uploading resource file:/home/myuser/.ivy2/jars/org.apache.avro_avro-1.7.6.jar -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.apache.avro_avro- 1.7.6.jar 18/06/13 16:00:26 INFO Client: Uploading resource file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-core-asl-1.9.13.jar -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org .codehaus.jackson_jackson-core-asl-1.9.13.jar 18/06/13 16:00:26 INFO Client: Uploading resource file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/o rg.codehaus.jackson_jackson-mapper-asl-1.9.13.jar 18/06/13 16:00:26 INFO Client: Uploading resource file:/home/myuser/.ivy2/jars/com.thoughtworks.paranamer_paranamer-2.3.jar -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/com.tho ughtworks.paranamer_paranamer-2.3.jar 18/06/13 16:00:26 INFO Client: Uploading resource file:/home/myuser/.ivy2/jars/org.xerial.snappy_snappy-java-1.0.5.jar -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.xerial.s nappy_snappy-java-1.0.5.jar 18/06/13 16:00:26 INFO Client: Uploading resource file:/home/myuser/.ivy2/jars/org.apache.commons_commons-compress-1.4.1.jar -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.ap ache.commons_commons-compress-1.4.1.jar 18/06/13 16:00:26 INFO Client: Uploading resource file:/home/myuser/.ivy2/jars/org.tukaani_xz-1.0.jar -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.tukaani_xz-1.0.jar 18/06/13 16:00:26 INFO Client: Source and destination file systems are the same. Not copying hdfs:/user/myuser/release/alphagenspark.zip#ROOT 18/06/13 16:00:26 INFO Client: Uploading resource file:/my/script/dir/spark/alphagen/foo.py -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/foo.py 18/06/13 16:00:26 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/pyspark.zip -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/pyspark.zip 18/06/13 16:00:26 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/py4j-0.10.4-src .zip 18/06/13 16:00:26 WARN Client: Same path resource file:/home/myuser/.ivy2/jars/com.databricks_spark-avro_2.11-4.0.0.jar added multiple times to distributed cache. 18/06/13 16:00:26 WARN Client: Same path resource file:/home/myuser/.ivy2/jars/org.slf4j_slf4j-api-1.7.5.jar added multiple times to distributed cache. 18/06/13 16:00:26 WARN Client: Same path resource file:/home/myuser/.ivy2/jars/org.apache.avro_avro-1.7.6.jar added multiple times to distributed cache. 18/06/13 16:00:26 WARN Client: Same path resource file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-core-asl-1.9.13.jar added multiple times to distributed cache. 18/06/13 16:00:26 WARN Client: Same path resource file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar added multiple times to distributed cache. 18/06/13 16:00:26 WARN Client: Same path resource file:/home/myuser/.ivy2/jars/com.thoughtworks.paranamer_paranamer-2.3.jar added multiple times to distributed cache. 18/06/13 16:00:26 WARN Client: Same path resource file:/home/myuser/.ivy2/jars/org.xerial.snappy_snappy-java-1.0.5.jar added multiple times to distributed cache. 18/06/13 16:00:26 WARN Client: Same path resource file:/home/myuser/.ivy2/jars/org.apache.commons_commons-compress-1.4.1.jar added multiple times to distributed cache. 18/06/13 16:00:26 WARN Client: Same path resource file:/home/myuser/.ivy2/jars/org.tukaani_xz-1.0.jar added multiple times to distributed cache. 18/06/13 16:00:27 INFO Client: Uploading resource file:/tmp/spark-6c26ae3b-7248-488f-bc33-9766251474bb/__spark_conf__4405623606341803690.zip -> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/__spark_conf__.zip 18/06/13 16:00:27 INFO SecurityManager: Changing view acls to: myuser 18/06/13 16:00:27 INFO SecurityManager: Changing modify acls to: myuser 18/06/13 16:00:27 INFO SecurityManager: Changing view acls groups to: 18/06/13 16:00:27 INFO SecurityManager: Changing modify acls groups to: 18/06/13 16:00:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(myuser); groups with view permissions: Set(); users with modify permissions: Set(myuser); groups with modify permissions: Set() 18/06/13 16:00:27 INFO Client: Submitting application application_1528901858967_0019 to ResourceManager 18/06/13 16:00:27 INFO YarnClientImpl: Submitted application application_1528901858967_0019 18/06/13 16:00:28 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:28 INFO Client: client token: N/A diagnostics: AM container is launched, waiting for AM container to Register with RM ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1528923627110 final status: UNDEFINED tracking URL: http://myhost2.myfirm.com:8088/proxy/application_1528901858967_0019/ user: myuser 18/06/13 16:00:29 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:30 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:31 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:32 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:33 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:34 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:35 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:36 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:37 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:38 INFO Client: Application report for application_1528901858967_0019 (state: ACCEPTED) 18/06/13 16:00:39 INFO Client: Application report for application_1528901858967_0019 (state: FAILED) 18/06/13 16:00:39 INFO Client: client token: N/A diagnostics: Application application_1528901858967_0019 failed 2 times due to AM Container for appattempt_1528901858967_0019_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://myhost2.myfirm.com:8088/cluster/app/application_1528901858967_0019 Then click on links to logs of each attempt. Diagnostics: java.util.zip.ZipException: invalid CEN header (bad signature) Failing this attempt. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1528923627110 final status: FAILED tracking URL: http://myhost2.myfirm.com:8088/cluster/app/application_1528901858967_0019 user: myuser 18/06/13 16:00:39 INFO Client: Deleted staging directory hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019 Exception in thread "main" org.apache.spark.SparkException: Application application_1528901858967_0019 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:1187) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1233) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 18/06/13 16:00:39 INFO ShutdownHookManager: Shutdown hook called 18/06/13 16:00:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-6c26ae3b-7248-488f-bc33-9766251474bb Has anyone seen this before? ... View more