Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

[HIVE concurrency request] erreur when run several same request

Labels:
avatar
author-rank kairel
New Contributor

Created ‎06-23-2021 01:28 AM

Hi all,

 

When we run multiple same request ( not exactly ) we have this exception:

 

 

 

Error: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] (state=,code=0)

 

 

 

In log file

 

 

 

 WARN  [HiveServer2-Handler-Pool: Thread-164113]: thrift.ThriftCLIService (:()) - Error fetching results: 
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]
	at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:478) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:952) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:565) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:792) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_112]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:602) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	... 13 more
Caused by: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:200) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:268) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:311) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:596) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:207) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:162) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:124) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:569) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	... 13 more
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) ~[?:1.8.0_112]
	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:320) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:182) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:268) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:311) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:596) ~[hbase-client-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:207) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:133) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:219) ~[hbase-mapreduce-2.0.2.3.1.4.0-315.jar:2.0.2.3.1.4.0-315]
	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:162) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:124) ~[hive-hbase-handler-3.1.0.3.1.4.0-315.jar:4.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:569) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2739) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
	... 13 more

 

 

 

 

I try to find this option "hive.server2.async.exec.threads" (maybe setted to 1), but i don't find in conf files

I'm not able to understand the issue, maybe a configuration issue ?

 

Im' able to reproduce the issue when a i run multiple same request via beeline client

3,862 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank kairel
New Contributor

Created ‎06-25-2021 12:25 AM

Thanks for your reply @ChethanYM ,

 

But it's not work..

 

My colleague have find the solution , it's another key to modify:

 

hive.server2.parallel.ops.in.session = true

 

Michael

View solution in original post

3,830 Views
0 Kudos
2 REPLIES 2

avatar
author-rank ChethanYM author-rank
Master Collaborator

Created ‎06-23-2021 07:21 AM

Hi @kairel 

This is hitting the HS2 thread count limit ,can you follow below steps and let us know if this helps?

 

Go ahead and set the following parameters:

 

1. Hive > Configs > Advanced > Custom hive-site 

     hive.server2.async.exec.threads=10

     hive.server2.async.exec.wait.queue.size=10

 

2.  Restart hiveserver2

3,846 Views
0 Kudos

avatar
author-rank kairel
New Contributor

Created ‎06-25-2021 12:25 AM

Thanks for your reply @ChethanYM ,

 

But it's not work..

 

My colleague have find the solution , it's another key to modify:

 

hive.server2.parallel.ops.in.session = true

 

Michael

3,831 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements