Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark2-Submit through Oozie Shell Action fails and multiple Spark application gets launched

Highlighted

Spark2-Submit through Oozie Shell Action fails and multiple Spark application gets launched

New Contributor

I have a Oozie WF in which I am doing spark2-submit via Shell Action. But the shell action get fails and in turn the Oozie WF gets failed with below mention exception:

 

Job failed as tasks failed. failedMaps:1 failedReduces:0

 

 

After some more debugging found the below mentioned exception in Application Master Logs:

 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

 

 

However the Oozie workflow fails but the Spark Application are up & running behind the scene and along with this multiple instances of same Spark Application gets launched.

 

Tried running the same Spark Application from CLI and it run fine with only 1 instance of job running.

 

Can anyone help me in resolving out this issue.

4 REPLIES 4
Highlighted

Re: Spark2-Submit through Oozie Shell Action fails and multiple Spark application gets launched

Expert Contributor

Can you attach the oozie launcher application logs?

 

Highlighted

Re: Spark2-Submit through Oozie Shell Action fails and multiple Spark application gets launched

New Contributor

Please find the logs below:

 

Dump of the process-tree for container_e109_1585210670345_1594_01_000005 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 31240 31238 31240 31240 (bash) 0 0 115847168 355 /bin/bash -c /usr/java/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.net.preferIPv4Stack=true -Xmx820m -Djava.io.tmpdir=/data11/yarn/nm/usercache/vishal/appcache/application_1585210670345_1594/container_e109_1585210670345_1594_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data02/yarn/container-logs/application_1585210670345_1594/container_e109_1585210670345_1594_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 172.23.80.79 41548 attempt_1585210670345_1594_m_000000_3 119846767427589 1>/data02/yarn/container-logs/application_1585210670345_1594/container_e109_1585210670345_1594_01_000005/stdout 2>/data02/yarn/container-logs/application_1585210670345_1594/container_e109_1585210670345_1594_01_000005/stderr  
	|- 31330 31248 31240 31240 (SparkETL.sh) 0 0 115851264 353 /bin/bash ./SparkETL.sh 20200331_09_25 /data_driven/workflows/config/dim_imei_WF.properties 2020-03-31T09:24Z jdbc:oracle:thin:@//172.19.131.58:1521 HDPAUXDW HDPAUXAPPO fk9hkfk9hk PROCESS_AUDIT 20g 5 10 
	|- 31689 31330 31240 31240 (java) 2302 153 35351339008 183609 /usr/java/jdk1.8.0_121/bin/java -cp /opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/conf/:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/jars/*:/data11/yarn/nm/usercache/vishal/appcache/application_1585210670345_1594/container_e109_1585210670345_1594_01_000005/oozie-hadoop-conf-1585646814247/:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/kafka-0.9/*:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/activation-1.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/aopalliance-1.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/apacheds-i18n-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/api-asn1-api-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/api-util-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/asm-3.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/avro-1.7.6-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/aws-java-sdk-bundle-1.11.134.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/azure-data-lake-store-sdk-2.2.9.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-beanutils-1.9.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-beanutils-core-1.8.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-codec-1.4.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-codec-1.9.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-configuration-1.6.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-daemon-1.0.13.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-digester-1.8.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-el-1.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-logging-1.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-math-2.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-math3-3.1.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/commons-net-3.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/core-3.1.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/curator-client-2.7.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/curator-framework-2.7.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/curator-recipes-2.7.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/disruptor-3.3.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/findbugs-annotations-1.3.9-1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/guava-12.0.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/guice-3.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-annotations-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-ant-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-archive-logs-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-archives-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-auth-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-aws-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-azure-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-azure-datalake-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-common-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-datajoin-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-distcp-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-extras-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-gridmix-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-hdfs-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-hdfs-nfs-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-mapreduce-client-app-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-mapreduce-client-common-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-mapreduce-client-core-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-mapreduce-client-hs-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-mapreduce-client-nativetask-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-mapreduce-examples-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-nfs-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-openstack-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-rumen-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-sls-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-streaming-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-api-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-applications-distributedshell-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-client-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-common-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-registry-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-server-common-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-server-nodemanager-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hadoop-yarn-server-web-proxy-2.6.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hamcrest-core-1.3.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-annotations-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-client-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-common-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-examples-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-external-blockcache-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-hadoop-compat-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-hadoop2-compat-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-it-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-prefix-tree-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-procedure-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-protocol-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-resource-bundle-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-rest-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-rsgroup-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-server-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-shell-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hbase-thrift-1.2.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/high-scale-lib-1.1.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hsqldb-1.8.0.10.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/htrace-core-3.2.0-incubating.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/htrace-core4-4.0.1-incubating.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/httpclient-4.2.5.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/httpcore-4.2.5.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hue-plugins-3.9.0-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jackson-annotations-2.2.3.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jackson-core-2.2.3.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jackson-core-asl-1.8.10.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jackson-databind-2.2.3-cloudera.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jackson-jaxrs-1.8.10.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jackson-mapper-asl-1.8.10-cloudera.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jackson-xc-1.8.10.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jamon-runtime-2.4.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jasper-compiler-5.5.23.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jasper-runtime-5.5.23.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/java-xmlbuilder-0.4.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/javax.inject-1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jaxb-api-2.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jaxb-api-2.2.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jaxb-impl-2.2.3-1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jcodings-1.0.8.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jets3t-0.9.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jettison-1.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jettison-1.3.3.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jline-2.11.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/joni-2.1.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jruby-cloudera-1.0.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jsch-0.1.42.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jsp-2.1-6.1.14.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jsp-api-2.1-6.1.14.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jsp-api-2.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jsr305-3.0.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/leveldbjni-all-1.8.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/metrics-core-2.2.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/metrics-core-3.0.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/microsoft-windowsazure-storage-sdk-0.6.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/mockito-all-1.8.5.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/netty-3.10.5.Final.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/netty-all-4.0.50.Final.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/okhttp-2.4.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/okio-1.4.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/paranamer-2.3.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/slf4j-log4j12-1.7.5.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/snappy-java-1.0.4.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/spark-1.6.0-cdh5.16.2-yarn-shuffle.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/spymemcached-2.11.6.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/stax-api-1.0-2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/xercesImpl-2.9.1.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/xml-apis-1.3.04.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/xmlenc-0.52.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/zookeeper-3.4.5-cdh5.16.2.jar:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/LICENSE.txt:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/lib/hadoop/NOTICE.txt:/usr/java/jdk1.8.0_121/lib/tools.jar org.apache.spark.deploy.SparkSubmit --master yarn --deploy-mode cluster --conf spark.driver.memory=6g --conf spark.speculation.quantile=0.90 --conf spark.driver.maxResultSize=5g --conf spark.sql.parquet.int96TimestampConversion=true --conf spark.executor.memoryOverhead=4096 --conf spark.speculation.multiplier=5 --conf spark.speculation=true --class co.truecorp.launcher.ETLLauncher --name dim_imei --queue cvm.dim_imei --executor-memory 20g --executor-cores 5 --num-executors 10 --jars hdfs://HDPPR1-NNHA/data_driven/workflows/jar/ojdbc8.jar,hdfs://HDPPR1-NNHA/data_driven/workflows/jar/orai18n.jar hdfs://HDPPR1-NNHA/data_driven/workflows/jar/Framework.jar yarn Table None None CVM_DATAMART IMEI dim_imei dim_imei_schema.csv dim_imei.properties hdfs://HDPPR1-NNHA/data_driven/edw/ raw/ curated/ 60 Yes No None None 20200331_09_25 30 jdbc:oracle:thin:@//172.19.131.58:1521 HDPAUXDW PROCESS_AUDIT HDPAUXAPPO fk9hkfk9hk 2020-03-31T09:24Z 2 Truncate 
	|- 31248 31240 31240 31240 (java) 1041 94 2801741824 78595 /usr/java/jdk1.8.0_121/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.net.preferIPv4Stack=true -Xmx820m -Djava.io.tmpdir=/data11/yarn/nm/usercache/vishal/appcache/application_1585210670345_1594/container_e109_1585210670345_1594_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data02/yarn/container-logs/application_1585210670345_1594/container_e109_1585210670345_1594_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 172.23.80.79 41548 attempt_1585210670345_1594_m_000000_3 119846767427589 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

2020-03-31 16:27:36,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1585210670345_1594_m_000000_3 TaskAttempt Transitioned from RUNNING to FAILED
2020-03-31 16:27:36,679 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e109_1585210670345_1594_01_000005 taskAttempt attempt_1585210670345_1594_m_000000_3
2020-03-31 16:27:36,688 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1585210670345_1594_m_000000 Task Transitioned from RUNNING to FAILED
2020-03-31 16:27:36,689 INFO [Thread-54] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node hdpndappr5.true.care
2020-03-31 16:27:36,689 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
2020-03-31 16:27:36,689 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:1 failedReduces:0
2020-03-31 16:27:36,691 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1585210670345_1594Job Transitioned from RUNNING to FAIL_ABORT
2020-03-31 16:27:36,693 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT
2020-03-31 16:27:36,705 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1585210670345_1594Job Transitioned from FAIL_ABORT to FAILED
2020-03-31 16:27:36,706 INFO [Thread-80] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2020-03-31 16:27:36,706 INFO [Thread-80] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2020-03-31 16:27:36,706 INFO [Thread-80] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
2020-03-31 16:27:36,706 INFO [Thread-80] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2020-03-31 16:27:36,706 INFO [Thread-80] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2020-03-31 16:27:36,706 INFO [Thread-80] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2020-03-31 16:27:36,707 INFO [Thread-80] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
2020-03-31 16:27:36,767 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://HDPPR1-NNHA:8020/user/vishal/.staging/job_1585210670345_1594/job_1585210670345_1594_1.jhist to hdfs://HDPPR1-NNHA:8020/user/history/done_intermediate/vishal/job_1585210670345_1594-1585646730642-vishal-oozie%3Alauncher%3AT%3Dshell%3AW%3DDIM_IMEI%3AA%3Dshell%2D9e79%3AID%3D-1585646856689-0-0-FAILED-root.users.vishal-1585646739236.jhist_tmp
2020-03-31 16:27:36,803 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://HDPPR1-NNHA:8020/user/history/done_intermediate/vishal/job_1585210670345_1594-1585646730642-vishal-oozie%3Alauncher%3AT%3Dshell%3AW%3DDIM_IMEI%3AA%3Dshell%2D9e79%3AID%3D-1585646856689-0-0-FAILED-root.users.vishal-1585646739236.jhist_tmp
2020-03-31 16:27:36,806 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://HDPPR1-NNHA:8020/user/vishal/.staging/job_1585210670345_1594/job_1585210670345_1594_1_conf.xml to hdfs://HDPPR1-NNHA:8020/user/history/done_intermediate/vishal/job_1585210670345_1594_conf.xml_tmp
2020-03-31 16:27:36,844 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://HDPPR1-NNHA:8020/user/history/done_intermediate/vishal/job_1585210670345_1594_conf.xml_tmp
2020-03-31 16:27:36,853 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://HDPPR1-NNHA:8020/user/history/done_intermediate/vishal/job_1585210670345_1594.summary_tmp to hdfs://HDPPR1-NNHA:8020/user/history/done_intermediate/vishal/job_1585210670345_1594.summary
2020-03-31 16:27:36,855 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://HDPPR1-NNHA:8020/user/history/done_intermediate/vishal/job_1585210670345_1594_conf.xml_tmp to hdfs://HDPPR1-NNHA:8020/user/history/done_intermediate/vishal/job_1585210670345_1594_conf.xml
2020-03-31 16:27:36,857 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://HDPPR1-NNHA:8020/user/history/done_intermediate/vishal/job_1585210670345_1594-1585646730642-vishal-oozie%3Alauncher%3AT%3Dshell%3AW%3DDIM_IMEI%3AA%3Dshell%2D9e79%3AID%3D-1585646856689-0-0-FAILED-root.users.vishal-1585646739236.jhist_tmp to hdfs://HDPPR1-NNHA:8020/user/history/done_intermediate/vishal/job_1585210670345_1594-1585646730642-vishal-oozie%3Alauncher%3AT%3Dshell%3AW%3DDIM_IMEI%3AA%3Dshell%2D9e79%3AID%3D-1585646856689-0-0-FAILED-root.users.vishal-1585646739236.jhist
2020-03-31 16:27:36,857 INFO [Thread-80] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2020-03-31 16:27:36,866 INFO [Thread-80] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to Task failed task_1585210670345_1594_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
Highlighted

Re: Spark2-Submit through Oozie Shell Action fails and multiple Spark application gets launched

Expert Contributor

The one that is provided doesn't include the actual mapper logs. May I know how you collected this log?

 

Can you run below command and provide the logs

yarn logs -applicationId application_1585210670345_1594 

Re: Spark2-Submit through Oozie Shell Action fails and multiple Spark application gets launched

Cloudera Employee

Hi,

 

Container killed on request. Exit code is 143 is a generic error and in order to find the root cause we need to see the application logs for that particular job where the container is getting error. 

 

Could you please share the application logs to identify the RCA for this issue?

 

Thanks

AKR

Don't have an account?
Coming from Hortonworks? Activate your account here