Confusing metrics for HBase in Cloudera Manager

[#1]

Hi,

I was digging in the HBase metrics available in Cloudera Manager and I can't really understand the differences between the following metrics:

• total_read_requests_rate_across_regionservers, total_write_requests_rate_across_regionservers (from the service home page) lists: 18000 req/s for reads - 100 req/s for writes
• total_requests_rate_across_regionservers (from the charts library): lists something like 1000 req/sec

Given the names I was expecting something like: total_requests = total_read_requests + total_write_requests but this is clearly not the case.

Which metric reflects the actual load of the HBase cluster?

Thanks,

parnigot.

[#2 · Accepted solution]

A bit of info:

- total_read_requests_rate_across_regionservers
tracks the RS JMX bean of Server::readRequestCount
- total_write_requests_rate_across_regionservers
tracks the RS JMX bean of Server::writeRequestCount
- total_requests_rate_across_regionservers
tracks the RS JMX bean of Server::totalRequestCount

The first two apply only to RS operations that operate on data, but the third applies also to other meta-operations such as openRegion, closeRegion, etc. that the RegionServer services (for Master and other commanding clients).

> Which metric reflects the actual load of the HBase cluster?

Data-wise its the read/write requests you want to look at.

> Given the names I was expecting something like: total_requests = total_read_requests + total_write_requests but this is clearly not the case.

The readRequestCount tracks only read operations (get/scan), where it also counts up multiple rows counted during scans. The totalRequestCount only counts by one per RPC done to RS, not per-row of reads done. This would cause a difference between the three metrics.

Hope this helps explain what these three metrics truly are. TL;DR:

total_read_requests_rate_across_regionservers -> Read operations count rate, counted per row scanned
total_write_requests_rate_across_regionservers -> Write operations count rate, counted per row written
total_requests_rate_across_regionservers -> Overall RS RPC-level call count rate, counted per request made to RS, not row-level