129793ms is a very long time indeed. I would suspect that the kernel is blocking the process and not scheduling it for some reason. I would look into kernel messages (like dmsg, /var/log/messages) etc for clues. ... View more

You can do this: - Manually kill the server server05 by issuing kill -9. This will cause the master to recognize that the server is dead, and will re-assign the regions that were hosted there. Also you can safely restart the master in a production env. Nothing in Hbase client depends on master being available, in regular read / write paths (only DDL statements). Master is pretty light and will come up quikly, so you can restart masters safely. ... View more

How many disks you have per node? Can you please check the region server logs about compaction related logging? Everytime a compaction runs and finishes, there should be some log saying how much data is compacted and how long it took etc. There seems to be something misconfigured in your environment. ... View more

ZooKeeper has SSL with Netty. But I am not sure it is tested well. https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide @Josh, in case of HBase tokens, I think they are stored in ZK. Can this be a concern? ... View more

You should close the Connection / Admin / Table WHEN you are done. According to #3, Connection creation is very costly, so you would want to share the connection as much as possible, which means that you won't be "done" with the Connection until the application shuts down. Close the Connection when you know that you won't be doing any more requests. However, Table and Admin is relatively cheap, so you should open / close those per request. ... View more

Thanks. Fixed the link. ... View more

You should run HBCK with -fix argument(s). ... View more

HBase in HDP is certified to run on top of HDFS, WASB/ADLS (using Azure HDInsight) and EMC Isilon. ... View more