Hbase read latency metrics shows 200000000ms . How to reduce this latency

[#1]

One of my cluster shows hbase read latency of 200000000ms which is impacting cluster performance and hence certain jobs. Need to resolve this as early as possible.

[#2]

Hi, could you provide more details like the slow regionserver's log and the hmaster's log? By the way, is there any RIT regions in hmaster's UI page?

[#3]

That's a latency of 55hrs or 2.3days for a read. That seems extremely unlikely that HBase would even be functioning if that's the case as the default RPC timeout for HBase is on the order of a few minutes. It seems like that graph may be inaccurate.

[#4]

I dont see any RIT on HBase webui but I see read requests of 7M & 3M on few datanodes. Attaching logs. Is it advisable to restart ambari-server in this case

[#5]

gclog-data02.txtgclog-data06.txt

[#6]

Hi @Anshul Sisodia, can you try to restart Ambari Metrics, and if you just did some Ambari config changes then also restart HBase. Also, I see that you have many minor GC compactions, they are fast but happen too often. OTOH, you have allocated only 4G for your RS, and have about 50G free on both RS nodes. Can you ramp them up to 16G, or at least 8G, and increase New gen to at least 1G. Either way, that latency looks wrong.

[#7 · Accepted solution]

Thanks for the input guys. I increased RS heap to 16GB and increased handler count to 200. Also we performed manual major compaction on few HBase tables majorly used by customer. After that HBase read performance dropped to acceptable limits.