Support Questions
Find answers, ask questions, and share your expertise

Insert Query into Hive gives exeption -- Tez related issue

Insert Query into Hive gives exeption -- Tez related issue

New Contributor

we have a Dev and prod - NiFi cluster - Version

The insert query executes fine on the Dev cluster but giving the below error on prod. I searched for this issue and looks like we need to disable TEZ and use MR as the execution engine. But why does it work in Dev where we have the same nifi version.

Also, can I disable TEZ in the hive connection pool using controller services.

I tried below 2 options :

a . set it in the property Database Connection URL as ${db_conn_url};hive.execution.engine=mr;

NiFi does not seem to be using this property.

b. add another property with hive.execution.engine as key and mr as value, but Nifi cannot validate it.

due to java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer 2, vertexId=vertex_1542468414407_6223_2_01, diagnostics=[Task failed, taskId=task_1542468414407_6223_2_01_000001, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) [Error getting row data with exception java.lang.ArrayIndexOutOfBoundsException: 54 at org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString( at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup( at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector( at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord( at at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor( at at at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$ at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$ at Method) at at at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal( at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal( at at at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec