Reply
Highlighted
Expert Contributor
Posts: 162
Registered: ‎07-29-2013
Accepted Solution

Confusing metrics for HBase requests from Cloudera Manager

Hi, hbase-master says that I have 1300-1500 requests per second for the whole cluster

Cloudera Manager 5.2.1 says that I have:

4500 read requests per second

500 write request per second.

 

I see that graphic on HBase service page.

 

Who Is lying?

 

Cloudera Employee
Posts: 576
Registered: ‎01-20-2014

Re: Confusing metrics for HBase requests from Cloudera Manager

Could you please check if either or both the HBase Canary (within HBase)
and HBase Region Health Canary (in Service Monitor) are turned on? I
suspect the Canary reads are what is skewing the stats. How many regions do
you have overall?

Regards,
Gautam Gopalakrishnan
Cloudera Support
Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: Confusing metrics for HBase requests from Cloudera Manager

Hi, thanks for the reply

 

I found these properties in HBase service configuration:

Enable HBase Canary=true

HBase Region Health Canary=true

 

>(in Service Monitor)

Didn't find there someting related to HBase

 

Cloudera Employee
Posts: 576
Registered: ‎01-20-2014

Re: Confusing metrics for HBase requests from Cloudera Manager

The most likely cause is that the HBase Canary (part of HBase, not from Service Monitor) is pinging each region's health regularly.

Prior to CM5.3.2, turning this HBase Canary off didn't work. So you could either
- upgrade to CM 5.3.2 (or later). This should not lead to cluster downtime as only CM and agents will be upgraded.
- lower the frequency of the the HBase Canary so that the discrepancy in stats is reduced.
Regards,
Gautam Gopalakrishnan
Cloudera Support
Announcements