Thank you! Will watch. ... View more

The above resting state Read Latency in our system was cut by 98% to 192 ms by increasing hbase.client.scanner.max.result.size from a default of 2 MB to 50 MB. In our case, we have HBASE 'cell size' (==Maximum Record Size == hbase.client.keyvalue.maxsize) of 10MB which is 5x larger than said default. hbase.client.scanner.max.result.size can be used to change the default "chunk size" transited back to the client. Again, by default this value is 2 MB since HBase 1.1+. Note about hbase.server.scanner.max.result.size — This setting enforces a maximum result size (in bytes), when reached the server will return the results is has so far. This is a safety setting and should be kept large. The default is inifinite in 0.98 and 1.0.x and 100mb in 1.1 and later. ... View more

Yes, Apache Thrift really needs to emit a useful error msg in the above scenario. ... View more

I'm seeing a similar "Read Latency" of 10 seconds in Ambari but with HBASE at rest (i.e., nobody is running any scans or anything with it). This was observed after loading ~30 GB of data (HDFS 30% full now) into a three-node HBase cluster having two Region Servers (these RS's had finished auto balancing regions between them followed by me executing a manual compaction from HBase webGUI). Even restarting HBase has no effect on read latency. I noticed too that Grafana has (a corresponding?) graph : HBase-Performance --> Operation Latencies : Scan Next -- but oddly it's one of the very few graphs labeled, "No datapoints", as if it were unplugged. Read latency goes up to 32 seconds when running 'hbase shell scan tableX'... no selective filter used. I'm surprised Ambari doesn't have a built in alarm that trips on anything over 2 seconds (Ambari GUI reports 'no alarms' even when read latency is ~18 min!) Pretty sure we have a config problem but not sure what. I'm running all this in AWS/Ubuntu14 using Ambari 2.2.2.0 (HDP 2.4.2.0-258) managed HBASE 1.1.2.2.4.2.0-258. ... View more

Thank you Josh! That was it. ... View more