Hbase Region servers not connecting Hbase master server in HDP2.6.3

[#1]

1. 2017-04-1318:13:38,481 WARN [regionserver/172.16.3.197:16020] regionserver.HRegionServer: error telling master we are up
2. com.google.protobuf.ServiceException: java.io.IOException:Call to abcdsolutions.local/172.16.3.196:16000 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:Call id=9, waitTime=10001, operationTimeout=10000 expired.
3. at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223)
4. at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
5. at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:10859)
6. at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2406)
7. at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:936)
8. at java.lang.Thread.run(Thread.java:745)
9. Causedby: java.io.IOException:Call to abcdsolutions.local/172.16.3.196:16000 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:Call id=9, waitTime=10001, operationTimeout=10000 expired.
10. at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1261)
11. at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
12. at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
13. ...5 more
14. Causedby: org.apache.hadoop.hbase.ipc.CallTimeoutException:Call id=9, waitTime=10001, operationTimeout=10000 expired.
15. at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
16. at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
17. ...6 more

[#2]

@raj pati

Error "Causedby: org.apache.hadoop.hbase.ipc.CallTimeoutException:Call id=9, waitTime=10001, operationTimeout=10000 expired." is a timeout exception.

Whats the value of hbase.rpc.timeout ?

hbase.client.scanner.timeout.period is a timeout specifically for RPCs that come from the HBase Scanner classes (e.g. ClientScanner) while hbase.rpc.timeout is the default timeout for any RPC. I believe that the hbase.client.scanner.timeout.period is also used by the RegionServers to define the lifetime of the Lease (the cause of the LeaseException you're seeing).

Generally, when you see these kinds of exceptions while scanning data in HBase, it is just a factor of your hardware and current performance (in other words, how long it takes to read your data). I can't really give a firm answer because it is dependent on your system's performance Could you adjust these parameters and restart the Hbase stale configs and test

Change the below values through Ambari and test

 <property> 
<name>hbase.client.scanner.timeout.period</name> 
<value>70000</value> 
</property> 

And also

<property> 
<name>hbase.rpc.timeout</name> 
<value>70000</value> 
</property> 

It should run successfully.