Support Questions

Find answers, ask questions, and share your expertise

Confusing metrics for HBase requests from Cloudera Manager

avatar
Expert Contributor

Hi, hbase-master says that I have 1300-1500 requests per second for the whole cluster

Cloudera Manager 5.2.1 says that I have:

4500 read requests per second

500 write request per second.

 

I see that graphic on HBase service page.

 

Who Is lying?

 

1 ACCEPTED SOLUTION

avatar
The most likely cause is that the HBase Canary (part of HBase, not from Service Monitor) is pinging each region's health regularly.

Prior to CM5.3.2, turning this HBase Canary off didn't work. So you could either
- upgrade to CM 5.3.2 (or later). This should not lead to cluster downtime as only CM and agents will be upgraded.
- lower the frequency of the the HBase Canary so that the discrepancy in stats is reduced.
Regards,
Gautam Gopalakrishnan

View solution in original post

3 REPLIES 3

avatar
Could you please check if either or both the HBase Canary (within HBase)
and HBase Region Health Canary (in Service Monitor) are turned on? I
suspect the Canary reads are what is skewing the stats. How many regions do
you have overall?

Regards,
Gautam Gopalakrishnan

avatar
Expert Contributor

Hi, thanks for the reply

 

I found these properties in HBase service configuration:

Enable HBase Canary=true

HBase Region Health Canary=true

 

>(in Service Monitor)

Didn't find there someting related to HBase

 

avatar
The most likely cause is that the HBase Canary (part of HBase, not from Service Monitor) is pinging each region's health regularly.

Prior to CM5.3.2, turning this HBase Canary off didn't work. So you could either
- upgrade to CM 5.3.2 (or later). This should not lead to cluster downtime as only CM and agents will be upgraded.
- lower the frequency of the the HBase Canary so that the discrepancy in stats is reduced.
Regards,
Gautam Gopalakrishnan