Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Querying hbase meta gets slow

Querying hbase meta gets slow

New Contributor

Scanning hbase:meta suddenly starts to get slower over time. Our main indicator is "Get_99th_percentile" for the node that holds meta, exposed via JMX. In normal operations it is below 10ms, but after 24-48 hours it starts to increase until we reach 50ms. This is when our clients start to re-establish the connection, and fail on pre-defined timeouts.

A workaround is to simply move meta to any other node. Though all look the same load- and hardware-wise, and have an uptime which is comparable, this solves the problem for the next app. 2 days. Then it starts to happen again.

Logging GC, which is already using G1, didn't unveil any insights so far.

Our stack:

HBase Version1.
Hadoop Version2., revision=13debf893a605e8a88df18a7d8d214f571e05289
Don't have an account?
Coming from Hortonworks? Activate your account here