Member since
09-29-2022
16
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2110 | 11-25-2022 09:34 PM |
03-21-2023
10:41 PM
Hi Team, I am getting below mentioned ERROR, could some one please suggest me how to fix it. Thanks in advance for your support always. job still getting failed - ERROR - AnalysisException raised in the UPSS_PROMO_PROMOTIONS Spark JOB Traceback (most recent call last): File "/hdp_bi_code/hdp_bi_code/upss/transformation/promo_promotions.py", line 203, in process_promotions scd2_history_data_latest.write.mode("overwrite").insertInto(targetDbMergeTableNm, overwrite=True) File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 716, in insertInto self._jwrite.mode("overwrite" if overwrite else "append").insertInto(tableName) File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in _call_ answer, self.gateway_client, self.target_id, self.name) File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/utils.py", line 69, in deco raise AnalysisException(s.split(': ', 1)[1], stackTrace) AnalysisException: u'java.lang.OutOfMemoryError: GC overhead limit exceeded;'
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Kafka
-
Apache Spark
03-03-2023
03:16 AM
Hi Team, Hadoop job got failed and getting below error. Need your expertise support. Thanks in advance for your kind support. ERROR - Unknown Exception:07<class 'py4j.protocol.Py4JJavaError'>An error occurred while calling o2068.saveAsTable. : org.apache.spark.SparkException: Job aborted. at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:224) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:154) at org.apache.spark.sql.execution.datasources.DataSource.writeAndRead(DataSource.scala:532) at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.saveDataIntoTable(createDataSourceTables.scala:216) at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:176) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102) at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127) at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80) at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:664) at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:664) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:664) at org.apache.spark.sql.DataFrameWriter.createTable(DataFrameWriter.scala:466) at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:445) at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:393) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748)
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Spark
01-31-2023
12:26 AM
Team, My current Apache spark jobs are executing very slow, could you please help me to make it Run Faster. Looking for thumb rule to make spark job it RUN FASTER. Thanks in advance for your kind support always.
... View more
Labels:
- Labels:
-
Apache Spark
01-22-2023
10:01 PM
Hi Team, It would be appreciated if someone please guide me how to set spark memory for spark job, where cluster utilization should take 1%-2% memory only for each spark job. Please share math's logic how to calculate on below cluster node details as - #1 How many working Nodes Cluster we have currently? | >Nodemanagers:166 >Datanodes:159 #2 How many Cores per Node we have currently ? | >64 Cores #3 How much GB RAM per node. we have currently ? | >503 GB ==== Wanted to calcuate for spark job === #1driver-memory #2 executor-memory #3 driver-cores #4 executor-cores #5 num-executor ======================== Please suggest if any additional parameter help to tune the spark job (execution time and cluster utilization) wise.
... View more
Labels:
- Labels:
-
Apache Spark
01-12-2023
12:33 AM
Hi All, Need some guideline for "how to define/design Dynamic SPARK job" any parameter. Please suggest me, it would be more appreciate in advance. Regards, Pankaj
... View more
Labels:
- Labels:
-
Apache Spark
12-13-2022
11:25 PM
I am trying to retrive hadoop table "select * from stg_db_kafka. limit 5 ;" betting the given error SQL Error: java.io.IOException: java.lang.RuntimeException: hdfs://NDC3HDPPROD/warehouse/tablespace/external/hive/stg_db_kafka.db/stg_prepaid_rated_usage_kafka/partitioncolumn=20220823085257/part-00002-7afffe68-eccb-4f9b-8f98-ac0a182c8e26.c000.snappy.parquet is not a Parquet file (too small length: 4) Could some one help me to fix it.
... View more
Labels:
- Labels:
-
Apache Kafka
11-25-2022
09:34 PM
GM Asish... Thanks for sharing difference between two parameter. Will take note on tez.task.resource.memory.mb should be set < hive.tez.container.size Appreciate for your support. Regards, Pankaj shivankar
... View more
11-25-2022
02:05 AM
okay . any reason if i include in task ?
... View more
11-25-2022
12:23 AM
Hi Asish, I was trying to add below parameter to do workaround memory. Will include new parameter mentioned by you for today's task and share update. Hope below parameter gives some optimized memory for Reason code - 2 solution. set tez.am.resource.memory.mb=16384; set tez.task.resource.memory.mb=16384; set hive.tez.container.size=16384; Thanks for sharing parameter with values. set tez.am.resource.memory.mb=10240; set tez.task.resource.memory.mb=10240; set tez.runtime.io.sort.mb=4096; set hive.tez.container.size=10240;
... View more
11-24-2022
01:51 AM
Hi All, Could you please help to resolve below concern as - We are executing below script and getting ERROR. I have enclosed the same for referene. hive -e "set hive.merge.tezfiles=true; set hive.merge.mapfiles=true; set hive.merge.mapredfiles=true; set tez.queue.name=BIAdv; set hive.execution.engine=tez; set hive.vectorized.execution.enabled =true; set hive.vectorized.execution.reduce.enabled ==true; set hive.exec.dynamic.partition=true;set hive.exec.dynamic.partition.mode=nonstrict; set hive.exec.max.dynamic.partitions.pernode=20000; set hive.exec.max.dynamic.partitions=100000;set hive.merge.size.per.task=134217724; set hive.merge.smallfiles.avgsize=134217724; set tez.grouping.split-count=1; INSERT overwrite TABLE <table-name1> partition (reported_date,last_usage_date) SELECT * from <table-name1> where reported_date='2022-11-16' and last_usage_date='2022-04-10';" ERROR - Log : ---------------------------------------------------------------------------------------------- ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1668260139179_120620_1_00, diagnostics=[Task failed, taskId=task_1668260139179_120620_1_00_000002, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnNames(ColumnProjectionUtils.java:239) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:163) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:964) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:409) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) , errorMessage=Cannot recover from this error:java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnNames(ColumnProjectionUtils.java:239) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:163) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:964) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:409) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:6, Vertex vertex_1668260139179_120620_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE] ERROR : Vertex killed, vertexName=Reducer 2, vertexId=vertex_1668260139179_120620_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex vertex_1668260139179_120620_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE] ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 INFO : org.apache.tez.common.counters.DAGCounter: INFO : NUM_FAILED_TASKS: 1 INFO : NUM_KILLED_TASKS: 6 INFO : NUM_SUCCEEDED_TASKS: 7 INFO : TOTAL_LAUNCHED_TASKS: 14 INFO : OTHER_LOCAL_TASKS: 2 INFO : RACK_LOCAL_TASKS: 5 INFO : AM_CPU_MILLISECONDS: 381090 INFO : AM_GC_TIME_MILLIS: 1830 INFO : File System Counters: INFO : FILE_BYTES_READ: 339136 INFO : FILE_BYTES_WRITTEN: 4844738 INFO : HDFS_BYTES_READ: 4005665433 INFO : HDFS_BYTES_WRITTEN: 1590596979 INFO : HDFS_READ_OPS: 71884 INFO : HDFS_WRITE_OPS: 2638 INFO : HDFS_OP_CREATE: 1183 INFO : HDFS_OP_GET_FILE_STATUS: 5887 INFO : HDFS_OP_MKDIRS: 279 INFO : HDFS_OP_OPEN: 65997 INFO : HDFS_OP_RENAME: 1176 INFO : org.apache.tez.common.counters.TaskCounter: INFO : SPILLED_RECORDS: 1176 INFO : GC_TIME_MILLIS: 3749370 INFO : TASK_DURATION_MILLIS: 10490948 INFO : CPU_MILLISECONDS: 35432800 INFO : PHYSICAL_MEMORY_BYTES: 46170898432 INFO : VIRTUAL_MEMORY_BYTES: 65302401024 INFO : COMMITTED_HEAP_BYTES: 46170898432 INFO : INPUT_RECORDS_PROCESSED: 46000210 INFO : INPUT_SPLIT_LENGTH_BYTES: 3020210217 INFO : OUTPUT_RECORDS: 1176 INFO : OUTPUT_LARGE_RECORDS: 0 INFO : OUTPUT_BYTES: 8054528 INFO : OUTPUT_BYTES_WITH_OVERHEAD: 8065910 INFO : OUTPUT_BYTES_PHYSICAL: 4675170 INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 0 INFO : ADDITIONAL_SPILLS_BYTES_READ: 0 INFO : ADDITIONAL_SPILL_COUNT: 0 INFO : SHUFFLE_CHUNK_COUNT: 7 INFO : HIVE: INFO : CREATED_DYNAMIC_PARTITIONS: 110 INFO : CREATED_FILES: 1176 INFO : DESERIALIZE_ERRORS: 0 INFO : RECORDS_IN_Map_1: 46000210 INFO : RECORDS_OUT_1_dim_cd_db.dim_vas_ppu_subs_base: 46000210 INFO : RECORDS_OUT_INTERMEDIATE_Map_1: 1176 INFO : RECORDS_OUT_OPERATOR_FS_3: 46000210 INFO : RECORDS_OUT_OPERATOR_GBY_6: 1176 INFO : RECORDS_OUT_OPERATOR_MAP_0: 0 INFO : RECORDS_OUT_OPERATOR_RS_7: 1176 INFO : RECORDS_OUT_OPERATOR_SEL_2: 46000210 INFO : RECORDS_OUT_OPERATOR_SEL_5: 46000210 INFO : RECORDS_OUT_OPERATOR_TS_0: 46000210 INFO : TaskCounter_Map_1_INPUT_dim_vas_ppu_subs_base: INFO : INPUT_RECORDS_PROCESSED: 46000210 INFO : INPUT_SPLIT_LENGTH_BYTES: 3020210217 INFO : TaskCounter_Map_1_OUTPUT_Reducer_2: INFO : ADDITIONAL_SPILLS_BYTES_READ: 0 INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 0 INFO : ADDITIONAL_SPILL_COUNT: 0 INFO : OUTPUT_BYTES: 8054528 INFO : OUTPUT_BYTES_PHYSICAL: 4675170 INFO : OUTPUT_BYTES_WITH_OVERHEAD: 8065910 INFO : OUTPUT_LARGE_RECORDS: 0 INFO : OUTPUT_RECORDS: 1176 INFO : SHUFFLE_CHUNK_COUNT: 7 INFO : SPILLED_RECORDS: 1176 INFO : org.apache.hadoop.hive.ql.exec.tez.HiveInputCounters: INFO : GROUPED_INPUT_SPLITS_Map_1: 14 INFO : INPUT_DIRECTORIES_Map_1: 169 INFO : INPUT_FILES_Map_1: 412191 INFO : RAW_INPUT_SPLITS_Map_1: 412191 ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1668260139179_120620_1_00, diagnostics=[Task failed, taskId=task_1668260139179_120620_1_00_000002, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnNames(ColumnProjectionUtils.java:239) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:163) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:964) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:409) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) , errorMessage=Cannot recover from this error:java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnNames(ColumnProjectionUtils.java:239) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:163) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:964) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:409) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:6, Vertex vertex_1668260139179_120620_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1668260139179_120620_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1009, Vertex vertex_1668260139179_120620_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 INFO : Completed executing command(queryId=hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5); Time taken: 7826.447 seconds INFO : Compiling command(queryId=hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5): INSERT overwrite TABLE dim_cd_db.dim_vas_ppu_subs_base partition (reported_date,last_usage_date) SELECT * from dim_cd_db.dim_vas_ppu_subs_base where reported_date='2022-09-25' INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:dim_vas_ppu_subs_base.subs_msisdn, type:varchar(25), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.circle_id, type:varchar(4), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.subs_key, type:varchar(25), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.pre_post_ind, type:varchar(2), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.act_mdt_cdr_id_key, type:varchar(70), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.called_calling_number, type:varchar(25), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.short_code, type:varchar(25), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.discovery_bearer, type:varchar(6), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.service_id, type:varchar(30), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.service_sub_sub_type_id, type:varchar(30), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.content_partner_code, type:varchar(30), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.activation_price_amt, type:decimal(14,6), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.current_price_amt, type:decimal(14,6), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.activation_start_date, type:date, comment:null), FieldSchema(name:dim_vas_ppu_subs_base.activation_end_date, type:date, comment:null), FieldSchema(name:dim_vas_ppu_subs_base.last_status_updt_date, type:date, comment:null), FieldSchema(name:dim_vas_ppu_subs_base.load_timestamp, type:timestamp, comment:null), FieldSchema(name:dim_vas_ppu_subs_base.reported_date, type:date, comment:null), FieldSchema(name:dim_vas_ppu_subs_base.last_usage_date, type:date, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5); Time taken: 1.779 seconds INFO : Executing command(queryId=hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5): INSERT overwrite TABLE dim_cd_db.dim_vas_ppu_subs_base partition (reported_date,last_usage_date) SELECT * from dim_cd_db.dim_vas_ppu_subs_base where reported_date='2022-09-25' INFO : Query ID = hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5 INFO : Total jobs = 3 INFO : Launching Job 1 out of 3 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Subscribed to counters: [] for queryId: hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5 INFO : Tez session hasn't been created yet. Opening session INFO : Dag name: INSERT overwrite TAB...ed_date='2022-09-25' (Stage-1) INFO : Status: Running (Executing on YARN cluster with App id application_1668260139179_121442) ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1668260139179_121442_1_00, diagnostics=[Task failed, taskId=task_1668260139179_121442_1_00_000002, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnNames(ColumnProjectionUtils.java:239) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:163) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:964) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:409) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) , errorMessage=Cannot recover from this error:java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnNames(ColumnProjectionUtils.java:239) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:163) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:964) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:409) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:6, Vertex vertex_1668260139179_121442_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE] ERROR : Vertex killed, vertexName=Reducer 2, vertexId=vertex_1668260139179_121442_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:2, Vertex vertex_1668260139179_121442_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE] ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 INFO : org.apache.tez.common.counters.DAGCounter: INFO : NUM_FAILED_TASKS: 1 INFO : NUM_KILLED_TASKS: 8 INFO : NUM_SUCCEEDED_TASKS: 7 INFO : TOTAL_LAUNCHED_TASKS: 16 INFO : OTHER_LOCAL_TASKS: 3 INFO : RACK_LOCAL_TASKS: 4 INFO : AM_CPU_MILLISECONDS: 344940 INFO : AM_GC_TIME_MILLIS: 1404 INFO : File System Counters: INFO : FILE_BYTES_READ: 784 INFO : FILE_BYTES_WRITTEN: 4249582 INFO : HDFS_BYTES_READ: 4005665433 INFO : HDFS_BYTES_WRITTEN: 1590596809 INFO : HDFS_READ_OPS: 71884 INFO : HDFS_WRITE_OPS: 2658 INFO : HDFS_OP_CREATE: 1183 INFO : HDFS_OP_GET_FILE_STATUS: 5887 INFO : HDFS_OP_MKDIRS: 299 INFO : HDFS_OP_OPEN: 65997 INFO : HDFS_OP_RENAME: 1176 INFO : org.apache.tez.common.counters.TaskCounter: INFO : SPILLED_RECORDS: 1176 INFO : GC_TIME_MILLIS: 2605189 INFO : TASK_DURATION_MILLIS: 8536033 INFO : CPU_MILLISECONDS: 33374430 INFO : PHYSICAL_MEMORY_BYTES: 45646610432 INFO : VIRTUAL_MEMORY_BYTES: 65281306624 INFO : COMMITTED_HEAP_BYTES: 45646610432 INFO : INPUT_RECORDS_PROCESSED: 46000210 INFO : INPUT_SPLIT_LENGTH_BYTES: 3020733963 INFO : OUTPUT_RECORDS: 1176 INFO : OUTPUT_LARGE_RECORDS: 0 INFO : OUTPUT_BYTES: 8054528 INFO : OUTPUT_BYTES_WITH_OVERHEAD: 8059316 INFO : OUTPUT_BYTES_PHYSICAL: 4249190 INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 0 INFO : ADDITIONAL_SPILLS_BYTES_READ: 0 INFO : ADDITIONAL_SPILL_COUNT: 0 INFO : SHUFFLE_CHUNK_COUNT: 7 INFO : HIVE: INFO : CREATED_DYNAMIC_PARTITIONS: 114 INFO : CREATED_FILES: 1176 INFO : DESERIALIZE_ERRORS: 0 INFO : RECORDS_IN_Map_1: 46000210 INFO : RECORDS_OUT_1_dim_cd_db.dim_vas_ppu_subs_base: 46000210 INFO : RECORDS_OUT_INTERMEDIATE_Map_1: 1176 INFO : RECORDS_OUT_OPERATOR_FS_3: 46000210 INFO : RECORDS_OUT_OPERATOR_GBY_6: 1176 INFO : RECORDS_OUT_OPERATOR_MAP_0: 0 INFO : RECORDS_OUT_OPERATOR_RS_7: 1176 INFO : RECORDS_OUT_OPERATOR_SEL_2: 46000210 INFO : RECORDS_OUT_OPERATOR_SEL_5: 46000210 INFO : RECORDS_OUT_OPERATOR_TS_0: 46000210 INFO : TaskCounter_Map_1_INPUT_dim_vas_ppu_subs_base: INFO : INPUT_RECORDS_PROCESSED: 46000210 INFO : INPUT_SPLIT_LENGTH_BYTES: 3020733963 INFO : TaskCounter_Map_1_OUTPUT_Reducer_2: INFO : ADDITIONAL_SPILLS_BYTES_READ: 0 INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 0 INFO : ADDITIONAL_SPILL_COUNT: 0 INFO : OUTPUT_BYTES: 8054528 INFO : OUTPUT_BYTES_PHYSICAL: 4249190 INFO : OUTPUT_BYTES_WITH_OVERHEAD: 8059316 INFO : OUTPUT_LARGE_RECORDS: 0 INFO : OUTPUT_RECORDS: 1176 INFO : SHUFFLE_CHUNK_COUNT: 7 INFO : SPILLED_RECORDS: 1176 INFO : org.apache.hadoop.hive.ql.exec.tez.HiveInputCounters: INFO : GROUPED_INPUT_SPLITS_Map_1: 14 INFO : INPUT_DIRECTORIES_Map_1: 169 INFO : INPUT_FILES_Map_1: 412191 INFO : RAW_INPUT_SPLITS_Map_1: 412191 ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1668260139179_121442_1_00, diagnostics=[Task failed, taskId=task_1668260139179_121442_1_00_000002, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnNames(ColumnProjectionUtils.java:239) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:163) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:964) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:409) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) , errorMessage=Cannot recover from this error:java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnNames(ColumnProjectionUtils.java:239) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:163) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:964) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:409) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:6, Vertex vertex_1668260139179_121442_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1668260139179_121442_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:2, Vertex vertex_1668260139179_121442_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 INFO : Completed executing command(queryId=hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5); Time taken: 7380.715 seconds Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1668260139179_121442_1_00, diagnostics=[Task failed, taskId=task_1668260139179_121442_1_00_000002, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnNames(ColumnProjectionUtils.java:239) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:163) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:964) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:409) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) , errorMessage=Cannot recover from this error:java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumnNames(ColumnProjectionUtils.java:239) at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:163) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:964) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:409) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152) at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:426) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:6, Vertex vertex_1668260139179_121442_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1668260139179_121442_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:2, Vertex vertex_1668260139179_121442_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 (state=08S01,code=2) Closing: 0: jdbc:hive2://ndc3hdpprodmn05.vodafoneidea.com:2181,ndc3hdpprodmn06.vodafoneidea.com:2181,ndc3hdpprodmn07.vodafoneidea.com:2181/default;httpPath=cliservice;password=biuser2;principal=hive/_HOST@INROOT.IN;serviceDiscoveryMode=zooKeeper;transportMode=http;user=biuser2;zooKeeperNamespace=hiveserver2 [biuser2@ndc3hdpproden01 Sudhakar]$ You have new mail in /var/spool/mail/biuser2 [biuser2@ndc3hdpproden01 Sudhakar]$ cat merge_vas_ppu.out SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.1.5.0-152/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.1.5.0-152/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://ndc3hdpprodmn05.vodafoneidea.com:2181,ndc3hdpprodmn06.vodafoneidea.com:2181,ndc3hdpprodmn07.vodafoneidea.com:2181/default;httpPath=cliservice;password=biuser2;principal=hive/_HOST@INROOT.IN;serviceDiscoveryMode=zooKeeper;transportMode=http;user=biuser2;zooKeeperNamespace=hiveserver2 22/11/21 22:51:33 [main]: INFO jdbc.HiveConnection: Connected to NDC3HDPPRODMN06.vodafoneidea.com:10001 Connected to: Apache Hive (version 3.1.0.3.1.5.0-152) Driver: Hive JDBC (version 3.1.0.3.1.5.0-152) Transaction isolation: TRANSACTION_REPEATABLE_READ No rows affected (0.064 seconds) No rows affected (0.009 seconds) No rows affected (0.008 seconds) No rows affected (0.006 seconds) No rows affected (0.005 seconds) No rows affected (0.004 seconds) No rows affected (0.004 seconds) No rows affected (0.004 seconds) No rows affected (0.004 seconds) No rows affected (0.004 seconds) No rows affected (0.003 seconds) No rows affected (0.004 seconds) No rows affected (0.004 seconds) No rows affected (0.004 seconds) INFO : Compiling command(queryId=hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5): INSERT overwrite TABLE dim_cd_db.dim_vas_ppu_subs_base partition (reported_date,last_usage_date) SELECT * from dim_cd_db.dim_vas_ppu_subs_base where reported_date='2022-09-25' INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:dim_vas_ppu_subs_base.subs_msisdn, type:varchar(25), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.circle_id, type:varchar(4), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.subs_key, type:varchar(25), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.pre_post_ind, type:varchar(2), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.act_mdt_cdr_id_key, type:varchar(70), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.called_calling_number, type:varchar(25), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.short_code, type:varchar(25), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.discovery_bearer, type:varchar(6), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.service_id, type:varchar(30), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.service_sub_sub_type_id, type:varchar(30), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.content_partner_code, type:varchar(30), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.activation_price_amt, type:decimal(14,6), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.current_price_amt, type:decimal(14,6), comment:null), FieldSchema(name:dim_vas_ppu_subs_base.activation_start_date, type:date, comment:null), FieldSchema(name:dim_vas_ppu_subs_base.activation_end_date, type:date, comment:null), FieldSchema(name:dim_vas_ppu_subs_base.last_status_updt_date, type:date, comment:null), FieldSchema(name:dim_vas_ppu_subs_base.load_timestamp, type:timestamp, comment:null), FieldSchema(name:dim_vas_ppu_subs_base.reported_date, type:date, comment:null), FieldSchema(name:dim_vas_ppu_subs_base.last_usage_date, type:date, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5); Time taken: 1.636 seconds INFO : Executing command(queryId=hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5): INSERT overwrite TABLE dim_cd_db.dim_vas_ppu_subs_base partition (reported_date,last_usage_date) SELECT * from dim_cd_db.dim_vas_ppu_subs_base where reported_date='2022-09-25' INFO : Query ID = hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5 INFO : Total jobs = 3 INFO : Launching Job 1 out of 3 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Subscribed to counters: [] for queryId: hive_20221121225133_cda2cb8e-97d3-48be-a1df-c0cd86b00ac5 INFO : Tez session hasn't been created yet. Opening session INFO : Dag name: INSERT overwrite TAB...ed_date='2022-09-25' (Stage-1) INFO : Status: Running (Executing on YARN cluster with App id application_1668260139179_120620)
... View more
Labels:
- Labels:
-
Apache Hive