question Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency in Support Questions

Hbase read latency metrics shows 200000000ms . How to reduce this latency

anshul_sisodia — Sun, 18 Aug 2019 12:41:21 GMT

One of my cluster shows hbase read latency of 200000000ms which is impacting cluster performance and hence certain jobs. Need to resolve this as early as possible.

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

vxu — Fri, 01 Apr 2016 22:56:38 GMT

Hi, could you provide more details like the slow regionserver's log and the hmaster's log? By the way, is there any RIT regions in hmaster's UI page?

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

elserj — Fri, 01 Apr 2016 23:03:47 GMT

That's a latency of 55hrs or 2.3days for a read. That seems extremely unlikely that HBase would even be functioning if that's the case as the default RPC timeout for HBase is on the order of a few minutes. It seems like that graph may be inaccurate.

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

anshul_sisodia — Fri, 01 Apr 2016 23:47:17 GMT

I dont see any RIT on HBase webui but I see read requests of 7M & 3M on few datanodes. Attaching logs. Is it advisable to restart ambari-server in this case

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

anshul_sisodia — Sat, 02 Apr 2016 01:52:55 GMT

gclog-data02.txt gclog-data06.txt

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

pminovic — Sat, 02 Apr 2016 11:51:52 GMT

Hi @Anshul Sisodia, can you try to restart Ambari Metrics, and if you just did some Ambari config changes then also restart HBase. Also, I see that you have many minor GC compactions, they are fast but happen too often. OTOH, you have allocated only 4G for your RS, and have about 50G free on both RS nodes. Can you ramp them up to 16G, or at least 8G, and increase New gen to at least 1G. Either way, that latency looks wrong.

Re: Hbase read latency metrics shows 200000000ms . How to reduce this latency

anshul_sisodia — Tue, 05 Apr 2016 19:14:36 GMT

Thanks for the input guys. I increased RS heap to 16GB and increased handler count to 200. Also we performed manual major compaction on few HBase tables majorly used by customer. After that HBase read performance dropped to acceptable limits.