HBase LeaseException

[#1]

I am using Apache HBase version 1.4.10 on AWS EMR backed by S3 (Instead of HDFS). I am seeing org.apache.hadoop.hbase.regionserver.LeaseException: lease '1643040576606302576' does not exist exception at the client-side, whereas the HBase region server is seeing below warning:

09:14:50,868 WARN  [RpcServer.default.RWQ.Fifo.scan.handler=508,queue=252,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1598692459275,"responsesize":5275840,"method":"Scan","param":"scanner_id: 1643040576606302576 number_of_rows: 2147483647 close_scanner: false next_call_seq: 6 client_handles_partials: true client_handles_heartbea ","processingtimems":31592,"client":"10.0.39.37:57984","queuetimems":1,"class":"HRegionServer","scandetails":"table: test region: test,h\\x0AMc\\xA0ts\\xE5\\xE3,1598322123108.89f04526e4ff17c7062b760b10acabac."}

09:15:55,411 WARN  [RpcServer.default.RWQ.Fifo.scan.handler=420,queue=215,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1598692490976,"responsesize":5275988,"method":"Scan","param":"scanner_id: 1643040576606302576 number_of_rows: 2147483647 close_scanner: false next_call_seq: 7 client_handles_partials: true client_handles_heartbea ","processingtimems":64435,"client":"10.0.39.37:58184","queuetimems":0,"class":"HRegionServer","scandetails":"table: test region: test,h\\x0AMc\\xA0ts\\xE5\\xE3,1598322123108.89f04526e4ff17c7062b760b10acabac."}

09:45:55,465 INFO  [regionserver/ip-10-0-17-66.ec2.internal/10.0.17.66:16020.leaseChecker] regionserver.RSRpcServices: Scanner 1643040576606302576 lease expired on region test,h\x0AMc\xA0ts\xE5\xE3,1598322123108.89f04526e4ff17c7062b760b10acabac.

Below is our HBase configuration, and we are using the same configuration for both the client and server-side. Despite it, I see the same issue. Sample major configuration related to this issue is as below:

<configuration>
   <property>
      <name>hbase.client.scanner.timeout.period</name>
      <value>1800000</value>
   </property>
   <property>
      <name>hbase.client.scanner.caching</name>
      <value>2147483647</value>
   </property>
   <property>
      <name>hbase.client.scanner.max.result.size</name>
      <value>5242880</value>
   </property>
   <property>
      <name>hbase.client.operation.timeout</name>
      <value>5640000</value>
   </property>
   <property>
      <name>hbase.rpc.timeout</name>
      <value>1860000</value>
   </property>
</configuration>

Below are application version details:

1. HBase Version : 1.4.10
2. ZooKeeper Client Version : 3.4.14
3. Hadoop Version : 2.8.5-amzn-5

Request:

Did anyone has seen the similar issue? Any suggestion?

[#2]

Hello @amit_ 

The Post is couple of months old, yet wish to know if you have fixed the issue. The Lease seems to have expired after ~30 Minutes as per your Configuration. However, the Log shows a processingTime of ~30/~60 Seconds for ~5MB of Scan Size, so the Scanner Lease expiring after ~30 Minutes looks strange & appears to be caused by either the Client going away somehow or the processingTime being excessively slow. 

If you have fixed the issue, Will appreciate your feedback on the Post.

- Smarak