Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!
Solved Go to solution

[HIVE concurrency request] erreur when run several same request

Labels:
avatar
author-rank kairel
New Member

Created ‎06-23-2021 01:28 AM

Hi all,

 

When we run multiple same request ( not exactly ) we have this exception:

 

 

 

Error: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] (state=,code=0)

 

 

 

In log file

 

 

 

 WARN  [HiveServer2-Handler-Pool: Thread-164113]: thrift.ThriftCLIService (:()) - Error fetching results: 
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]
	at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:478) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:952) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:565) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:792) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_112]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:602) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	... 13 more
Caused by: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:200) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:268) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:311) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:596) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:207) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:162) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:124) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:569) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	... 13 more
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) ~[?:1.8.0_112]
	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:320) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:182) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:268) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:311) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:596) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:207) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:162) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:124) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:569) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	... 13 more

 

 

 

 

I try to find this option "hive.server2.async.exec.threads" (maybe setted to 1), but i don't find in conf files

I'm not able to understand the issue, maybe a configuration issue ?

 

Im' able to reproduce the issue when a i run multiple same request via beeline client

5,655 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank kairel
New Member

Created ‎06-25-2021 12:25 AM

Thanks for your reply @ChethanYM ,

 

But it's not work..

 

My colleague have find the solution , it's another key to modify:

 

hive.server2.parallel.ops.in.session = true

 

Michael

View solution in original post

5,623 Views
0 Kudos
2 REPLIES 2

avatar
author-rank ChethanYM author-rank
Master Collaborator

Created ‎06-23-2021 07:21 AM

Hi @kairel 

This is hitting the HS2 thread count limit ,can you follow below steps and let us know if this helps?

 

Go ahead and set the following parameters:

 

1. Hive > Configs > Advanced > Custom hive-site 

     hive.server2.async.exec.threads=10

     hive.server2.async.exec.wait.queue.size=10

 

2.  Restart hiveserver2

5,639 Views
0 Kudos

avatar
author-rank kairel
New Member

Created ‎06-25-2021 12:25 AM

Thanks for your reply @ChethanYM ,

 

But it's not work..

 

My colleague have find the solution , it's another key to modify:

 

hive.server2.parallel.ops.in.session = true

 

Michael

5,624 Views
0 Kudos
Announcements
Community Announcements
Community Upgrade Announcement
What's New @ Cloudera
Announcing Cloudera Flow Management Operator for Kubernetes ...
What's New @ Cloudera
Announcing Cloudera Streams Messaging Operator for Kubernete...
Community Announcements
July 2026 Community Highlights
Community Announcements
June 2026 Community Highlights
View More Announcements