Confusing metrics for HBase requests from Cloudera Manager

[#1]

Hi, hbase-master says that I have 1300-1500 requests per second for the whole cluster

Cloudera Manager 5.2.1 says that I have:

4500 read requests per second

500 write request per second.

I see that graphic on HBase service page.

Who Is lying?

[#2]

Could you please check if either or both the HBase Canary (within HBase)
and HBase Region Health Canary (in Service Monitor) are turned on? I
suspect the Canary reads are what is skewing the stats. How many regions do
you have overall?

[#3]

Hi, thanks for the reply

I found these properties in HBase service configuration:

Enable HBase Canary=true

HBase Region Health Canary=true

>(in Service Monitor)

Didn't find there someting related to HBase

[#4 · Accepted solution]

The most likely cause is that the HBase Canary (part of HBase, not from Service Monitor) is pinging each region's health regularly.

Prior to CM5.3.2, turning this HBase Canary off didn't work. So you could either
- upgrade to CM 5.3.2 (or later). This should not lead to cluster downtime as only CM and agents will be upgraded.
- lower the frequency of the the HBase Canary so that the discrepancy in stats is reduced.