Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive LLAP java.lang.AssertionError

Highlighted

Hive LLAP java.lang.AssertionError

New Contributor

Hi All,

I am hitting the below exception while running a SELECT statement in LLAP. Same query works on Tez..

The environment is Hive 1.2 and HDP 2.6.1. Any clues on what could be wrong?

2017-08-16 20:37:22,269 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1502817965713_0066_21][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 6, taskAttemptId=attempt_1502817965713_0066_21_04_000009_0, creationTime=1502915841055, allocationTime=1502915841247, startTime=1502915841248, finishTime=1502915842269, timeTaken=1021, status=FAILED, taskFailureType=FATAL, errorEnum=APPLICATION_ERROR, diagnostics=Error: Error while running task ( failure ) : org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.io.IOException: java.lang.AssertionError: Unexpected refCount -1: 0x5b87f493(-1)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.io.IOException: java.lang.AssertionError: Unexpected refCount -1: 0x5b87f493(-1)
	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
	at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
	at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
	at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
	at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
	... 17 more<br>
3 REPLIES 3
Highlighted

Re: Hive LLAP java.lang.AssertionError

@Sreejith Madhavan

The issue seems to be related to the HIVE-10161 when LLAP I/O is enabled and issue are seen wrt Buffer.
Try with hive.llap.io.enabled=false.

Highlighted

Re: Hive LLAP java.lang.AssertionError

New Contributor

Thanks for the pointers. As stated in the jira, restarting llap did not help, however, hive.llap.io.enabled=false did. Do we have a permanent fix for this issue other than setting the above flag?

Re: Hive LLAP java.lang.AssertionError

Rising Star

@Sreejith Madhavan I see the comment "running a SELECT statement in LLAP" but using Hive1.2 env, Try using Hive2 code base.

Connect to HSI (Hiveserver2Interactive) url via beeline, which will make use of llap containers

Don't have an account?
Coming from Hortonworks? Activate your account here