Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hbase Region servers not connecting Hbase master server in HDP2.6.3

Highlighted

Hbase Region servers not connecting Hbase master server in HDP2.6.3

New Contributor
  1. 2017-04-1318:13:38,481 WARN [regionserver/172.16.3.197:16020] regionserver.HRegionServer: error telling master we are up
  2. com.google.protobuf.ServiceException: java.io.IOException:Call to abcdsolutions.local/172.16.3.196:16000 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:Call id=9, waitTime=10001, operationTimeout=10000 expired.
  3. at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223)
  4. at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
  5. at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:10859)
  6. at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2406)
  7. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:936)
  8. at java.lang.Thread.run(Thread.java:745)
  9. Causedby: java.io.IOException:Call to abcdsolutions.local/172.16.3.196:16000 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:Call id=9, waitTime=10001, operationTimeout=10000 expired.
  10. at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1261)
  11. at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
  12. at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
  13. ...5 more
  14. Causedby: org.apache.hadoop.hbase.ipc.CallTimeoutException:Call id=9, waitTime=10001, operationTimeout=10000 expired.
  15. at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
  16. at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
  17. ...6 more
1 REPLY 1

Re: Hbase Region servers not connecting Hbase master server in HDP2.6.3

Mentor

@raj pati

Error "Causedby: org.apache.hadoop.hbase.ipc.CallTimeoutException:Call id=9, waitTime=10001, operationTimeout=10000 expired." is a timeout exception.

Whats the value of hbase.rpc.timeout ?

hbase.client.scanner.timeout.period is a timeout specifically for RPCs that come from the HBase Scanner classes (e.g. ClientScanner) while hbase.rpc.timeout is the default timeout for any RPC. I believe that the hbase.client.scanner.timeout.period is also used by the RegionServers to define the lifetime of the Lease (the cause of the LeaseException you're seeing).

Generally, when you see these kinds of exceptions while scanning data in HBase, it is just a factor of your hardware and current performance (in other words, how long it takes to read your data). I can't really give a firm answer because it is dependent on your system's performance Could you adjust these parameters and restart the Hbase stale configs and test

Change the below values through Ambari and test

 <property> 
<name>hbase.client.scanner.timeout.period</name> 
<value>70000</value> 
</property> 

And also

<property> 
<name>hbase.rpc.timeout</name> 
<value>70000</value> 
</property> 

It should run successfully.