Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created on
09-14-2021
04:18 AM
- last edited on
04-21-2026
01:58 AM
by
GrazittiAPI
On CDP we are using both the Hive (for the Hive Metastores) and Hive on Tez (for the HiveServers).
We are getting the below error while trying to run a query based on a condition. I can't share the table information and the exact query but it looks something as below.
CREATE EXTERNAL TABLE IF NOT EXISTS XXX (
`1` string,
`6` varchar(30),
`7` varchar(5),
`8` varchar(10)
) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES('hbase.columns.mapping'=':key,
xx:1,
xx:5,
xx:6')
TBLPROPERTIES (
'hbase.table.name'='YYYYY'
);
the query looks as follows:
select * from XXX where 8 = '1990-10-10';
And we see the below error from the HiveServer
[a3ed3b7b-d225-43af-9ac0-76917911a742 HiveServer2-Handler-Pool: Thread-128-EventThread]: Error while calling watcher
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d7573cd rejected from java.util.concurrent.ThreadPoolExecutor@194ae4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_242]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) ~[?:1.8.0_242]
at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) ~[?:1.8.0_242]
at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:541) ~[hbase-zookeeper-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) ~[hbase-zookeeper-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) [zookeeper-3.5.5.7.1.6.0-297.jar:3.5.5.7.1.6.0-297]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [zookeeper-3.5.5.7.1.6.0-297.jar:3.5.5.7.1.6.0-297]
We added the below config on the HiveServer (based on this: https://community.cloudera.com/t5/Support-Questions/HIVE-concurrency-request-erreur-when-run-several...) but still, we are getting the issue.
<property>
<name>hive.server2.parallel.ops.in.session</name>
<value>true</value>
</property>
Created 09-14-2021 10:34 PM
Can you set the property below and try to rerun the query.
set hive.fetch.task.conversion=none;
Created 09-14-2021 10:34 PM
Can you set the property below and try to rerun the query.
set hive.fetch.task.conversion=none;
Created on 09-15-2021 01:17 AM - edited 09-15-2021 01:18 AM
Thank you @balajip. I tried the solution. Updated the config on Hive on Tez, but still, I am getting the issue. The full stack trace is given below.
[HiveServer2-Handler-Pool: Thread-114015]: Error fetching results:
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:476) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:946) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:567) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:798) [hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:654) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:638) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:901) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:243) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:471) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
... 13 more
Caused by: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:200) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:267) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:310) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:595) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:211) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:140) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:101) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:605) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:901) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:243) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:471) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
... 13 more
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@4fcc96f rejected from java.util.concurrent.ThreadPoolExecutor@66bee5ac[Shutting down, pool size = 162, active threads = 0, queued tasks = 0, completed tasks = 5297]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_242]
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:329) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:191) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:267) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:310) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:595) ~[hbase-client-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:211) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:140) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:101) ~[hive-hbase-handler-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:605) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:901) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:243) ~[hive-exec-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:471) ~[hive-service-3.1.3000.7.1.6.0-297.jar:3.1.3000.7.1.6.0-297]
... 13 more
Created 09-15-2021 05:04 AM
@balajip I set the config in the wrong service. After setting the config on Hive service it worked. Thank you.