Hello,   Thanks for your responce, Where ever i am checking the HBCK on that particular server we have HBase master role and from Gateway node also i can see same error.   Any other suggessions wolud be appriciated.   Thanks. ... View more

Resurrecting this topic with some clarity on the issue and it's remedy.   If RegionServers would be keeping dead connections to the Datanodes, the same symnptoms would be seen, many connection in CLOSE_WAIT, and file descriptor number increasing. In extreme cases the limit could be reached, whioch would case the host node to fail with no more open file descriptors to use issue.   There wasa bug in HBase prior to CDH5.13 which is described in this upstream JIRA in more detail[1]: HBASE-9393 Hbase does not closing a closed socket resulting in many CLOSE_WAIT]   This issue was patched in the following CDH releases: CDH5.13.0, CDH5.13.1, CDH5.13.2, CDH5.13.3, CDH5.14.0,  CDH5.14.2, CDH5.14.4, CDH5.15.0,  CDH5.15.1, CDH6.0.0.   [1] - upstream HBase JIRA - https://issues.apache.org/jira/browse/HBASE-9393?attachmentOrder=asc ... View more

  I am just sharing the relevant part of the linked docs, as they contain the instructions on how to enable the hbase balancer via hbase shell:   Load Balancer It is assumed that the Region Load Balancer is disabled while the graceful_stop script runs (otherwise the balancer and the decommission script will end up fighting over region deployments). Use the shell to disable the balancer: hbase(main):001:0> balance_switch false true 0 row(s) in 0.3590 seconds This turns the balancer OFF. To reenable, do: hbase(main):001:0> balance_switch true false 0 row(s) in 0.3590 seconds The graceful_stop will check the balancer and if enabled, will turn it off before it goes to work. If it exits prematurely because of error, it will not have reset the balancer. Hence, it is better to manage the balancer apart from graceful_stopreenabling it after you are done w/ graceful_stop. ... View more