Hive Query Fails in Tez, Runs in MR mode

anilpallickal — Wed, 03 Oct 2018 02:08:50 GMT

Hive Query fails with the below error when running in Tez Mode. My application dont have option to alter the execution mode. So tez is the only option. Is this a bug from Tez ? Query run fine in MR mode when tried from shell..

Error:

-------------------------------------------------------------------------------- Map 1 ..... RUNNING 494 278 0 216 343 0 Reducer 2 INITED 62 0 0 62 0 0 -------------------------------------------------------------------------------- VERTICES: 00/02 [=============>>-------------] 50% ELAPSED TIME: 41.84 s -------------------------------------------------------------------------------- Status: Failed Vertex failed, vertexName=Map 1, vertexId=vertex_1537195062060_938252_1_00, diagnostics=[Task failed, taskId=task_1537195062060_938252_1_00_000367, diagn=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOEon: java.io.IOException: wrong key class: org.apache.hadoop.io.NullWritable is not class org.apache.hadoop.io.BytesWritable at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.io.IOException: wrong key class: org.apache.hadoop.io.NullWritablet class org.apache.hadoop.io.BytesWritable at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:71) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more

Re: Hive Query Fails in Tez, Runs in MR mode

krystle_salazar — Wed, 03 Oct 2018 05:48:06 GMT

Same problem here while triying to insert in the Sandbox HDP 2.6.4.

And the MapReduce mode is very slow too, it would be great be able to use Hive with TEZ.

Re: Hive Query Fails in Tez, Runs in MR mode

nramanaiah — Wed, 03 Oct 2018 15:01:17 GMT

Probably you might be hitting TEZ-2741.

can you try to run the query by setting below config

hive.compute.splits.in.am=false

Re: Hive Query Fails in Tez, Runs in MR mode

anilpallickal — Wed, 03 Oct 2018 23:38:02 GMT

Worked like a charm!. Thank you.

Re: Hive Query Fails in Tez, Runs in MR mode

Shelton — Thu, 04 Oct 2018 00:08:43 GMT

@Krystle Salazar

Set that on the hive CLI

set hive.executio.engine=tez;

Should do

question Re: Hive Query Fails in Tez, Runs in MR mode in Archives of Support Questions (Read Only)

Hive Query Fails in Tez, Runs in MR mode

Re: Hive Query Fails in Tez, Runs in MR mode

Re: Hive Query Fails in Tez, Runs in MR mode

Re: Hive Query Fails in Tez, Runs in MR mode

Re: Hive Query Fails in Tez, Runs in MR mode