Created 12-15-2016 04:30 PM
I'm having an intermittent timeout error when running some example code against HBase. The basic Java application creates a Scanner and queries a particular HBase table. For small queries, my code works fine. But, when I increase the TimeRange of my query, I get intermittent timeout errors, as seen below. Googling and searching the forum has not yielded any plausible solutions. Does anyone have any idea what the source of this error might be, and how to mitigate it?
Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions: [java] Thu Dec 15 10:33:43 EST 2016, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60304: row '' on table 'my-table-name' at region=my-table-name,,1481665174391.d068f4be09585cf831dbcd3a04664caf., hostname=hostname-007.localdomain.local,16020,1481747196976, seqNum=34514987 [java] [java] at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271) [java] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:195) [java] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59) [java] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) [java] at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320) [java] at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:403) [java] at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364) [java] at test.MyTestClass.main(MyTestClass.java:70) [java] Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60304: row '' on table 'my-table-name' at region=my-table-name,,1481665174391.d068f4be09585cf831dbcd3a04664caf., hostname=hostname-007.localdomain.local,16020,1481747196976, seqNum=34514987 [java] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159) [java] at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64) [java] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [java] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [java] at java.lang.Thread.run(Thread.java:745) [java] Caused by: java.io.IOException: Call to hostname-007.localdomain.local/10.0.0.106:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2, waitTime=60001, operationTimeout=60000 expired. [java] at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1262) [java] at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1230) [java] at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) [java] at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) [java] at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651) [java] at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213) [java] at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62) [java] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) [java] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:346) [java] at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:320) [java] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) [java] ... 4 more [java] Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2, waitTime=60001, operationTimeout=60000 expired. [java] at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70) [java] at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1204) [java] ... 13 more [java] Java Result: 1
Created 03-30-2017 09:54 AM
hello,i use spark hivecontext to read a hive-hbase-external table.then,it seems to that i got the same problem. how did you solve this problem? can you share the solution with me?
Created 10-29-2018 08:35 AM
Hello, @John O'Hara is this problem resolved ? Even I'm having the issue I have rpc timeout set as 90000 but I get the error as 60000.