Support Questions
Find answers, ask questions, and share your expertise

Hbase read latency metrics shows 200000000ms . How to reduce this latency

Solved Go to solution
Highlighted

Hbase read latency metrics shows 200000000ms . How to reduce this latency

Rising Star

One of my cluster shows hbase read latency of 200000000ms which is impacting cluster performance and hence certain jobs. Need to resolve this as early as possible.

3131-hbase-metricspic.jpg

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

Rising Star

Thanks for the input guys. I increased RS heap to 16GB and increased handler count to 200. Also we performed manual major compaction on few HBase tables majorly used by customer. After that HBase read performance dropped to acceptable limits.

View solution in original post

6 REPLIES 6

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

Contributor

Hi, could you provide more details like the slow regionserver's log and the hmaster's log? By the way, is there any RIT regions in hmaster's UI page?

Highlighted

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

That's a latency of 55hrs or 2.3days for a read. That seems extremely unlikely that HBase would even be functioning if that's the case as the default RPC timeout for HBase is on the order of a few minutes. It seems like that graph may be inaccurate.

Highlighted

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

Rising Star

I dont see any RIT on HBase webui but I see read requests of 7M & 3M on few datanodes. Attaching logs. Is it advisable to restart ambari-server in this case

Highlighted

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

Rising Star
Highlighted

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

Hi @Anshul Sisodia, can you try to restart Ambari Metrics, and if you just did some Ambari config changes then also restart HBase. Also, I see that you have many minor GC compactions, they are fast but happen too often. OTOH, you have allocated only 4G for your RS, and have about 50G free on both RS nodes. Can you ramp them up to 16G, or at least 8G, and increase New gen to at least 1G. Either way, that latency looks wrong.

Highlighted

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

Rising Star

Thanks for the input guys. I increased RS heap to 16GB and increased handler count to 200. Also we performed manual major compaction on few HBase tables majorly used by customer. After that HBase read performance dropped to acceptable limits.

View solution in original post