Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Confusing metrics for HBase in Cloudera Manager

Solved Go to solution

Confusing metrics for HBase in Cloudera Manager

Explorer

Hi,

 

I was digging in the HBase metrics available in Cloudera Manager and I can't really understand the differences between the following metrics:

 

  • total_read_requests_rate_across_regionservers, total_write_requests_rate_across_regionservers (from the service home page) lists: 18000 req/s for reads - 100 req/s for writes
  • total_requests_rate_across_regionservers (from the charts library): lists something like 1000 req/sec

Given the names I was expecting something like: total_requests = total_read_requests + total_write_requests but this is clearly not the case.

 

Which metric reflects the actual load of the HBase cluster?

 

Thanks,

parnigot.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Confusing metrics for HBase in Cloudera Manager

Master Guru
A bit of info:

- total_read_requests_rate_across_regionservers
tracks the RS JMX bean of Server::readRequestCount
- total_write_requests_rate_across_regionservers
tracks the RS JMX bean of Server::writeRequestCount
- total_requests_rate_across_regionservers
tracks the RS JMX bean of Server::totalRequestCount

The first two apply only to RS operations that operate on data, but the third applies also to other meta-operations such as openRegion, closeRegion, etc. that the RegionServer services (for Master and other commanding clients).

> Which metric reflects the actual load of the HBase cluster?

Data-wise its the read/write requests you want to look at.

> Given the names I was expecting something like: total_requests = total_read_requests + total_write_requests but this is clearly not the case.

The readRequestCount tracks only read operations (get/scan), where it also counts up multiple rows counted during scans. The totalRequestCount only counts by one per RPC done to RS, not per-row of reads done. This would cause a difference between the three metrics.

Hope this helps explain what these three metrics truly are. TL;DR:

total_read_requests_rate_across_regionservers -> Read operations count rate, counted per row scanned
total_write_requests_rate_across_regionservers -> Write operations count rate, counted per row written
total_requests_rate_across_regionservers -> Overall RS RPC-level call count rate, counted per request made to RS, not row-level
1 REPLY 1

Re: Confusing metrics for HBase in Cloudera Manager

Master Guru
A bit of info:

- total_read_requests_rate_across_regionservers
tracks the RS JMX bean of Server::readRequestCount
- total_write_requests_rate_across_regionservers
tracks the RS JMX bean of Server::writeRequestCount
- total_requests_rate_across_regionservers
tracks the RS JMX bean of Server::totalRequestCount

The first two apply only to RS operations that operate on data, but the third applies also to other meta-operations such as openRegion, closeRegion, etc. that the RegionServer services (for Master and other commanding clients).

> Which metric reflects the actual load of the HBase cluster?

Data-wise its the read/write requests you want to look at.

> Given the names I was expecting something like: total_requests = total_read_requests + total_write_requests but this is clearly not the case.

The readRequestCount tracks only read operations (get/scan), where it also counts up multiple rows counted during scans. The totalRequestCount only counts by one per RPC done to RS, not per-row of reads done. This would cause a difference between the three metrics.

Hope this helps explain what these three metrics truly are. TL;DR:

total_read_requests_rate_across_regionservers -> Read operations count rate, counted per row scanned
total_write_requests_rate_across_regionservers -> Write operations count rate, counted per row written
total_requests_rate_across_regionservers -> Overall RS RPC-level call count rate, counted per request made to RS, not row-level
Don't have an account?
Coming from Hortonworks? Activate your account here