[HIVE concurrency request] erreur when run several same request

kairel — Wed, 23 Jun 2021 08:28:40 GMT

Hi all,

When we run multiple same request ( not exactly ) we have this exception:

Error: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] (state=,code=0)

In log file

WARN [HiveServer2-Handler-Pool: Thread-164113]: thrift.ThriftCLIService (:()) - Error fetching results: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:478) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:952) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:565) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:792) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_112] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Caused by: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:602) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] ... 13 more Caused by: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:200) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:268) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:311) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:596) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:207) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:162) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT] at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:124) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:569) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] ... 13 more Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) ~[?:1.8.0_112] at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:320) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:182) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:268) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:311) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:596) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:207) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315] at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:162) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT] at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:124) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:569) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] ... 13 more

I try to find this option "hive.server2.async.exec.threads" (maybe setted to 1), but i don't find in conf files

I'm not able to understand the issue, maybe a configuration issue ?

Im' able to reproduce the issue when a i run multiple same request via beeline client

Re: [HIVE concurrency request] erreur when run several same request

ChethanYM — Wed, 23 Jun 2021 14:21:33 GMT

Hi @kairel

This is hitting the HS2 thread count limit ,can you follow below steps and let us know if this helps?

Go ahead and set the following parameters:

1. Hive > Configs > Advanced > Custom hive-site

hive.server2.async.exec.threads=10

hive.server2.async.exec.wait.queue.size=10

2. Restart hiveserver2

Re: [HIVE concurrency request] erreur when run several same request

kairel — Fri, 25 Jun 2021 07:25:02 GMT

Thanks for your reply @ChethanYM ,

But it's not work..

My colleague have find the solution , it's another key to modify:

hive.server2.parallel.ops.in.session = true

Michael

question Re: [HIVE concurrency request] erreur when run several same request in Support Questions

[HIVE concurrency request] erreur when run several same request

Re: [HIVE concurrency request] erreur when run several same request

Re: [HIVE concurrency request] erreur when run several same request