Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

Hive Error while calling watcher java.util.concurrent.RejectedExecutionException

avatar
author-rank Sayed016
Contributor

Created on ‎09-14-2021 04:18 AM - edited ‎09-14-2021 04:19 AM

On CDP we are using both the Hive (for the Hive Metastores) and Hive on Tez (for the HiveServers).

 

We are getting the below error while trying to run a query based on a condition. I can't share the table information and the exact query but it looks something as below.

 

CREATE EXTERNAL TABLE IF NOT EXISTS XXX (
    `1` string,
    `6` varchar(30),
    `7` varchar(5),
    `8` varchar(10)
) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES('hbase.columns.mapping'=':key,
    xx:1,
    xx:5,
    xx:6')
TBLPROPERTIES (
    'hbase.table.name'='YYYYY'
);

 

 

the query looks as follows:

 

select * from XXX where 8 = '1990-10-10';

 

 

And we see the below error from the HiveServer

 

[a3ed3b7b-d225-43af-9ac0-76917911a742 HiveServer2-Handler-Pool: Thread-128-EventThread]: Error while calling watcher 
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d7573cd rejected from java.util.concurrent.ThreadPoolExecutor@194ae4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_242]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_242]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_242]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) ~[?:1.8.0_242]
	at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) ~[?:1.8.0_242]
	at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:541) ~[hbase-zookeeper-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
	at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) ~[hbase-zookeeper-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) [zookeeper-3.5.5.7.1.6.0-297.jar:3.5.5.7.1.6.0-297]
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [zookeeper-3.5.5.7.1.6.0-297.jar:3.5.5.7.1.6.0-297]

 

 

We added the below config on the HiveServer (based on this: https://community.cloudera.com/t5/Support-Questions/HIVE-concurrency-request-erreur-when-run-several...) but still, we are getting the issue.

 

<property>
    <name>hive.server2.parallel.ops.in.session</name>
    <value>true</value>
</property>

 

 

 

 

 

5,153 Views
0 Kudos
0
1 ACCEPTED SOLUTION

avatar
author-rank balajip author-rank
Expert Contributor

Created ‎09-14-2021 10:34 PM

@Sayed016 

Can you set the property below and try to rerun the query.
set hive.fetch.task.conversion=none;

 

View solution in original post

5,122 Views
0
3 REPLIES 3

avatar
author-rank balajip author-rank
Expert Contributor

Created ‎09-14-2021 10:34 PM

@Sayed016 

Can you set the property below and try to rerun the query.
set hive.fetch.task.conversion=none;

 

5,123 Views
0

avatar
author-rank Sayed016
Contributor

Created on ‎09-15-2021 01:17 AM - edited ‎09-15-2021 01:18 AM

Thank you @balajip. I tried the solution. Updated the config on Hive on Tez, but still, I am getting the issue. The full stack trace is given below.

 

 

[HiveServer2-Handler-Pool: Thread-114015]: Error fetching results: 
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
    at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:476) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:946) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:567) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:798) [hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:654) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:638) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:901) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:243) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:471) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    ... 13 more
Caused by: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
    at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:200) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:267) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:310) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:595) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:211) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:140) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:101) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:605) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:901) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:243) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:471) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    ... 13 more
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_242]
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_242]
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_242]
    at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:329) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:191) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:267) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:310) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:595) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:211) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
    at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:140) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:101) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:605) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:901) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:243) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:471) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
    ... 13 more

 

 

 

 

5,108 Views
0 Kudos

avatar
author-rank Sayed016
Contributor

Created ‎09-15-2021 05:04 AM

@balajip I set the config in the wrong service. After setting the config on Hive service it worked. Thank you.

 

5,096 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics - Kubernetes Operato...
What's New @ Cloudera
[RELEASED] Cloudera Streams Messaging - Kubernetes Operator ...
Community Announcements
February 2025 Community Highlights
What's New @ Cloudera
3 Benefits of External IDE Connectivity, Now Available in Cl...
What's New @ Cloudera
Performance comparison of Spark3 on YARN with S3 Standard VS...
View More Announcements