Hive LLAP java.lang.AssertionError

[#1]

Hi All,

I am hitting the below exception while running a SELECT statement in LLAP. Same query works on Tez..

The environment is Hive 1.2 and HDP 2.6.1. Any clues on what could be wrong?

2017-08-16 20:37:22,269 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1502817965713_0066_21][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 6, taskAttemptId=attempt_1502817965713_0066_21_04_000009_0, creationTime=1502915841055, allocationTime=1502915841247, startTime=1502915841248, finishTime=1502915842269, timeTaken=1021, status=FAILED, taskFailureType=FATAL, errorEnum=APPLICATION_ERROR, diagnostics=Error: Error while running task ( failure ) : org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.io.IOException: java.lang.AssertionError: Unexpected refCount -1: 0x5b87f493(-1)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.io.IOException: java.lang.AssertionError: Unexpected refCount -1: 0x5b87f493(-1)
	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
	at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
	at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
	at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
	at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
	... 17 more<br>

[#2]

@Sreejith Madhavan

The issue seems to be related to the HIVE-10161 when LLAP I/O is enabled and issue are seen wrt Buffer.
Try with hive.llap.io.enabled=false.

[#3]

Thanks for the pointers. As stated in the jira, restarting llap did not help, however, hive.llap.io.enabled=false did. Do we have a permanent fix for this issue other than setting the above flag?

[#4]

@Sreejith Madhavan I see the comment "running a SELECT statement in LLAP" but using Hive1.2 env, Try using Hive2 code base.

Connect to HSI (Hiveserver2Interactive) url via beeline, which will make use of llap containers