Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

Hive Error while calling watcher java.util.concurrent.RejectedExecutionException

avatar
Contributor

On CDP we are using both the Hive (for the Hive Metastores) and Hive on Tez (for the HiveServers).

 

We are getting the below error while trying to run a query based on a condition. I can't share the table information and the exact query but it looks something as below.

 

CREATE EXTERNAL TABLE IF NOT EXISTS XXX (
    `1` string,
    `6` varchar(30),
    `7` varchar(5),
    `8` varchar(10)
) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES('hbase.columns.mapping'=':key,
    xx:1,
    xx:5,
    xx:6')
TBLPROPERTIES (
    'hbase.table.name'='YYYYY'
);

 

 

the query looks as follows:

 

select * from XXX where 8 = '1990-10-10';

 

 

And we see the below error from the HiveServer

 

[a3ed3b7b-d225-43af-9ac0-76917911a742 HiveServer2-Handler-Pool: Thread-128-EventThread]: Error while calling watcher 
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d7573cd rejected from java.util.concurrent.ThreadPoolExecutor@194ae4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_242]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_242]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_242]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) ~[?:1.8.0_242]
	at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) ~[?:1.8.0_242]
	at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:541) ~[hbase-zookeeper-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
	at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) ~[hbase-zookeeper-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) [zookeeper-3.5.5.7.1.6.0-297.jar:3.5.5.7.1.6.0-297]
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [zookeeper-3.5.5.7.1.6.0-297.jar:3.5.5.7.1.6.0-297]

 

 

We added the below config on the HiveServer (based on this: https://community.cloudera.com/t5/Support-Questions/HIVE-concurrency-request-erreur-when-run-several...) but still, we are getting the issue.

 

<property>
    <name>hive.server2.parallel.ops.in.session</name>
    <value>true</value>
</property>

 

 

 

 

 

1 ACCEPTED SOLUTION

avatar
Expert Contributor

@Sayed016 

Can you set the property below and try to rerun the query.
set hive.fetch.task.conversion=none;

 

View solution in original post

3 REPLIES 3

avatar
Expert Contributor

@Sayed016 

Can you set the property below and try to rerun the query.
set hive.fetch.task.conversion=none;

 

avatar
Contributor

Thank you @balajip. I tried the solution. Updated the config on Hive on Tez, but still, I am getting the issue. The full stack trace is given below.

 

 

[HiveServer2-Handler-Pool: Thread-114015]: Error fetching results: 
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
    at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:476) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:946) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:567) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:798) [hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:654) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:638) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:901) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:243) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:471) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    ... 13 more
Caused by: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
    at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:200) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:267) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:310) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:595) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:211) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:140) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:101) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:605) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:901) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:243) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:471) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    ... 13 more
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_242]
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_242]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_242]
    at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:329) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:191) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:267) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:310) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:595) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:211) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:140) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:101) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:605) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:901) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:243) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:471) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    ... 13 more

 

 

 

 

avatar
Contributor

@balajip I set the config in the wrong service. After setting the config on Hive service it worked. Thank you.