Support Questions
Find answers, ask questions, and share your expertise
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

Querying hbase meta gets slow

New Contributor

Scanning hbase:meta suddenly starts to get slower over time. Our main indicator is "Get_99th_percentile" for the node that holds meta, exposed via JMX. In normal operations it is below 10ms, but after 24-48 hours it starts to increase until we reach 50ms. This is when our clients start to re-establish the connection, and fail on pre-defined timeouts.

A workaround is to simply move meta to any other node. Though all look the same load- and hardware-wise, and have an uptime which is comparable, this solves the problem for the next app. 2 days. Then it starts to happen again.

Logging GC, which is already using G1, didn't unveil any insights so far.

Our stack:

HBase Version1.
Hadoop Version2., revision=13debf893a605e8a88df18a7d8d214f571e05289