Hive Runtime Error while processing row - Using TEZ

[#1]

Hello. We are receiving an error when trying to run some Hive queries, e.g. the one below:

select count(*) from test_mkt where newcolumn12

This query is executed via Tez using Hue in an Ambari Cluster. I've seen a couple of errors similar to this one, but none seems to be solved, at least in those threads. As an additional note, the general performance in Hue was quite poor and slow by the time this query was executed. Not sure if one issue is related to the other...

After more or less one hour of such low performance, Hue went back to normal and every queries since then were correctly executed (and no further issues appeared).

Below is the log file from the failed task:

Vertex failed, vertexName=Map 1, vertexId=vertex_1486479973965_0154_2_00, diagnostics=Task failed, taskId=task_1486479973965_0154_2_00_000000, diagnostics=TaskAttempt 0 failed, info=

Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:328) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Boolean at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:119) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552) ... 18 moreTaskAttempt 1 failed, info=Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:328) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Boolean at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:119) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552) ... 18 moreTaskAttempt 2 failed, info=

Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:328) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Boolean at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:119) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552) ... 18 moreTaskAttempt 3 failed, info=

Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:328) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"newcolumn":"ClientID","newcolumn1":"SendID","newcolumn2":"SubscriberKey","newcolumn3":"EmailAddress","newcolumn4":"SubscriberID","newcolumn5":"ListID","newcolumn6":"EventDate","newcolumn7":"EventType","newcolumn8":"BounceCategory","newcolumn9":"SMTPCode","newcolumn10":"BounceReason","newcolumn11":"BatchID","newcolumn12":"TriggeredSendExternalKey"} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Boolean at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:119) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:167) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552) ... 18 more Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1486479973965_0154_2_00Map 1 killed/failed due to:OWN_TASK_FAILURE Vertex killed, vertexName=Reducer 2, vertexId=vertex_1486479973965_0154_2_01, diagnostics=Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1486479973965_0154_2_01Reducer 2 killed/failed due to:OTHER_VERTEX_FAILURE

DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1

[#2]

@Juan Castrilli

The problem seems to be due to the error,

Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Boolean at

It is trying to convert string data type to boolean, since you have mentioned any comparison operator with the column (newcolumn12)

Query:

select count(*) from test_mkt where newcolumn12

Can you try with a valid where clause condition for this column data type.

ex:

select count(*) from test_mkt where newcolumn12='xyz';

[#3]

Thanks for your response, Rajesh! This could indeed be the issue. I'll ask the user to try and run the query using a valid clause as you mentioned and see if everything works out.