Solved
Hive Error while calling watcher java.util.concurrent.RejectedExecutionException
On CDP we are using both the Hive (for the Hive Metastores) and Hive on Tez (for the HiveServers).
We are getting the below error while trying to run a query based on a condition. I can't share the table information and the exact query but it looks something as below.
CREATE EXTERNAL TABLE IF NOT EXISTS XXX (
`1` string,
`6` varchar(30),
`7` varchar(5),
`8` varchar(10)
) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES('hbase.columns.mapping'=':key,
xx:1,
xx:5,
xx:6')
TBLPROPERTIES (
'hbase.table.name'='YYYYY'
);
the query looks as follows:
select * from XXX where 8 = '1990-10-10';
And we see the below error from the HiveServer
[a3ed3b7b-d225-43af-9ac0-76917911a742 HiveServer2-Handler-Pool: Thread-128-EventThread]: Error while calling watcher
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d7573cd rejected from java.util.concurrent.ThreadPoolExecutor@194ae4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_242]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) ~[?:1.8.0_242]
at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) ~[?:1.8.0_242]
at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:541) ~[hbase-zookeeper-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) ~[hbase-zookeeper-2.2.3.7.1.6.0-297.jar:2.2.3.7.1.6.0-297]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) [zookeeper-3.5.5.7.1.6.0-297.jar:3.5.5.7.1.6.0-297]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) [zookeeper-3.5.5.7.1.6.0-297.jar:3.5.5.7.1.6.0-297]
We added the below config on the HiveServer (based on this: https://community.cloudera.com/t5/Support-Questions/HIVE-concurrency-request-erreur-when-run-several-same/td-p/319166) but still, we are getting the issue.
<property>
<name>hive.server2.parallel.ops.in.session</name>
<value>true</value>
</property>
