Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Confusing metrics for HBase requests from Cloudera Manager

SOLVED Go to solution

Confusing metrics for HBase requests from Cloudera Manager

Expert Contributor

Hi, hbase-master says that I have 1300-1500 requests per second for the whole cluster

Cloudera Manager 5.2.1 says that I have:

4500 read requests per second

500 write request per second.

 

I see that graphic on HBase service page.

 

Who Is lying?

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Confusing metrics for HBase requests from Cloudera Manager

The most likely cause is that the HBase Canary (part of HBase, not from Service Monitor) is pinging each region's health regularly.

Prior to CM5.3.2, turning this HBase Canary off didn't work. So you could either
- upgrade to CM 5.3.2 (or later). This should not lead to cluster downtime as only CM and agents will be upgraded.
- lower the frequency of the the HBase Canary so that the discrepancy in stats is reduced.
Regards,
Gautam Gopalakrishnan
3 REPLIES 3

Re: Confusing metrics for HBase requests from Cloudera Manager

Could you please check if either or both the HBase Canary (within HBase)
and HBase Region Health Canary (in Service Monitor) are turned on? I
suspect the Canary reads are what is skewing the stats. How many regions do
you have overall?

Regards,
Gautam Gopalakrishnan
Highlighted

Re: Confusing metrics for HBase requests from Cloudera Manager

Expert Contributor

Hi, thanks for the reply

 

I found these properties in HBase service configuration:

Enable HBase Canary=true

HBase Region Health Canary=true

 

>(in Service Monitor)

Didn't find there someting related to HBase

 

Re: Confusing metrics for HBase requests from Cloudera Manager

The most likely cause is that the HBase Canary (part of HBase, not from Service Monitor) is pinging each region's health regularly.

Prior to CM5.3.2, turning this HBase Canary off didn't work. So you could either
- upgrade to CM 5.3.2 (or later). This should not lead to cluster downtime as only CM and agents will be upgraded.
- lower the frequency of the the HBase Canary so that the discrepancy in stats is reduced.
Regards,
Gautam Gopalakrishnan