Hi,
Cluster Version : Cloudera Express 5.14.3
No Recent changes since last 3 Months on the Cluster
Recently, a Week Back, I started getting the below Error and the Job Fails...Jobs are getting executed from Oozie.
Any Feedback ?
"ERROR : FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask"
"Error: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=3)"
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.exec.dynamic.partition.mode=nonstrict;
No rows affected (0.005 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.exec.max.dynamic.partitions=1000;
No rows affected (0.004 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.exec.max.dynamic.partitions.pernode=1000;
No rows affected (0.004 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.exec.compress.output=true;
No rows affected (0.004 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.exec.parallel=true;
No rows affected (0.005 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET hive.execution.engine=spark;
No rows affected (0.008 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET spark.executor.cores=1;
No rows affected (0.004 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET spark.executor.memory=3g;
No rows affected (0.005 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET spark.executor.memoryOverhead=1g;
No rows affected (0.003 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET spark.yarn.executor.memoryOverhead=1024;
No rows affected (0.004 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET avro.output.codec=snappy;
No rows affected (0.003 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SET mapreduce.job.queuename=batchlayer;
No rows affected (0.003 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> add jar hdfs:///user/admeta/udf/protobuf-hive-serde.jar;
INFO : converting to local hdfs:///user/admeta/udf/protobuf-hive-serde.jar
INFO : Added [/tmp/8ab71e5f-efbc-4c74-bccb-82e9475aa4e1_resources/protobuf-hive-serde.jar] to class path
INFO : Added resources: [hdfs:///user/admeta/udf/protobuf-hive-serde.jar]
No rows affected (0.221 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SELECT batchid, 1 FROM batch_state.import_to_event WHERE batchid IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201);
INFO : Compiling command(queryId=hive_20181004152727_2f987052-db1c-43e6-91ff-589fe3ea1d71): SELECT batchid, 1 FROM batch_state.import_to_event WHERE batchid IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201)
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:batchid, type:bigint, comment:null), FieldSchema(name:_c1, type:int, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20181004152727_2f987052-db1c-43e6-91ff-589fe3ea1d71); Time taken: 0.134 seconds
INFO : Executing command(queryId=hive_20181004152727_2f987052-db1c-43e6-91ff-589fe3ea1d71): SELECT batchid, 1 FROM batch_state.import_to_event WHERE batchid IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201)
INFO : Query ID = hive_20181004152727_2f987052-db1c-43e6-91ff-589fe3ea1d71
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in parallel
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : Starting Spark Job = 5978239f-723b-4a4b-bf7e-e474da7eb6f4
INFO : Running with YARN Application = application_1538413789685_4130
INFO : Kill Command = /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/bin/yarn application -kill application_1538413789685_4130
INFO :
Query Hive on Spark job[0] stages:
INFO : 0
INFO :
Status: Running (Hive on Spark job[0])
INFO : Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
INFO : 2018-10-04 15:27:43,845 Stage-0_0: 0(+1)/1
INFO : 2018-10-04 15:27:46,862 Stage-0_0: 0(+1)/1
INFO : 2018-10-04 15:27:49,878 Stage-0_0: 0(+1)/1
INFO : 2018-10-04 15:27:50,883 Stage-0_0: 1/1 Finished
INFO : Status: Finished successfully in 18.06 seconds
INFO : =====Spark Job[5978239f-723b-4a4b-bf7e-e474da7eb6f4] statistics=====
INFO : HIVE
INFO : CREATED_FILES: 1
INFO : RECORDS_OUT_0: 10
INFO : RECORDS_IN: 33056
INFO : DESERIALIZE_ERRORS: 0
INFO : Spark Job[5978239f-723b-4a4b-bf7e-e474da7eb6f4] Metrics
INFO : ExecutorDeserializeTime: 2022
INFO : ExecutorRunTime: 4244
INFO : ResultSize: 2107
INFO : JvmGCTime: 194
INFO : ResultSerializationTime: 1
INFO : MemoryBytesSpilled: 0
INFO : DiskBytesSpilled: 0
INFO : BytesRead: 11526
INFO : Execution completed successfully
INFO : Completed executing command(queryId=hive_20181004152727_2f987052-db1c-43e6-91ff-589fe3ea1d71); Time taken: 28.004 seconds
INFO : OK
+-------------------+------+--+
| batchid | _c1 |
+-------------------+------+--+
| 2018092513420201 | 1 |
| 2018092513480201 | 1 |
| 2018092513540101 | 1 |
| 2018092514000101 | 1 |
| 2018092514060101 | 1 |
| 2018092514120101 | 1 |
| 2018092514180101 | 1 |
| 2018092514240101 | 1 |
| 2018092514300201 | 1 |
| 2018092514360101 | 1 |
+-------------------+------+--+
10 rows selected (28.251 seconds)
0: jdbc:hive2://hadoop40.wopd.local:10000/def> SELECT
. . . . . . . . . . . . . . . . . . . . . . .> count(*)
. . . . . . . . . . . . . . . . . . . . . . .> FROM import.adserverevent ase
. . . . . . . . . . . . . . . . . . . . . . .> WHERE ase.BatchId IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201);
INFO : Compiling command(queryId=hive_20181004153030_3adbc088-a0f5-470d-a5ba-3c62091494d8): SELECT
count(*)
FROM import.adserverevent ase
WHERE ase.BatchId IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201)
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20181004153030_3adbc088-a0f5-470d-a5ba-3c62091494d8); Time taken: 1.17 seconds
INFO : Executing command(queryId=hive_20181004153030_3adbc088-a0f5-470d-a5ba-3c62091494d8): SELECT
count(*)
FROM import.adserverevent ase
WHERE ase.BatchId IN (2018092513480201, 2018092513420201, 2018092513540101, 2018092514240101, 2018092514120101, 2018092514360101, 2018092514000101, 2018092514060101, 2018092514180101, 2018092514300201)
INFO : Query ID = hive_20181004153030_3adbc088-a0f5-470d-a5ba-3c62091494d8
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in parallel
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : Starting Spark Job = a18751ee-97a0-4c5b-86e2-b87eeb1933dc
INFO : Running with YARN Application = application_1538413789685_4130
INFO : Kill Command = /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/bin/yarn application -kill application_1538413789685_4130
INFO :
Query Hive on Spark job[1] stages:
INFO : 1
INFO : 2
INFO :
Status: Running (Hive on Spark job[1])
INFO : Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
INFO : 2018-10-04 15:30:50,545 Stage-1_0: 0(+1)/9 Stage-2_0: 0/1
INFO : 2018-10-04 15:30:53,567 Stage-1_0: 0(+1)/9 Stage-2_0: 0/1
INFO : 2018-10-04 15:30:54,573 Stage-1_0: 0(+2)/9 Stage-2_0: 0/1
INFO : 2018-10-04 15:30:57,594 Stage-1_0: 0(+6)/9 Stage-2_0: 0/1
INFO : 2018-10-04 15:30:58,601 Stage-1_0: 0(+7)/9 Stage-2_0: 0/1
INFO : 2018-10-04 15:31:01,666 Stage-1_0: 0(+8)/9 Stage-2_0: 0/1
INFO : 2018-10-04 15:31:04,707 Stage-1_0: 0(+8)/9 Stage-2_0: 0/1
INFO : 2018-10-04 15:31:05,713 Stage-1_0: 0(+9)/9 Stage-2_0: 0/1
INFO : 2018-10-04 15:31:08,731 Stage-1_0: 0(+9)/9 Stage-2_0: 0/1
INFO : 2018-10-04 15:31:11,750 Stage-1_0: 0(+9)/9 Stage-2_0: 0/1
INFO : 2018-10-04 15:31:13,767 Stage-1_0: 1(+8)/9 Stage-2_0: 0/1
ERROR : Status: Failed
ERROR : FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
INFO : Completed executing command(queryId=hive_20181004153030_3adbc088-a0f5-470d-a5ba-3c62091494d8); Time taken: 30.006 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=3)
0: jdbc:hive2://hadoop40.wopd.local:10000/def>
