Resurrecting this topic with some clarity on the issue and it's remedy.   If RegionServers would be keeping dead connections to the Datanodes, the same symnptoms would be seen, many connection in CLOSE_WAIT, and file descriptor number increasing. In extreme cases the limit could be reached, whioch would case the host node to fail with no more open file descriptors to use issue.   There wasa bug in HBase prior to CDH5.13 which is described in this upstream JIRA in more detail[1]: HBASE-9393 Hbase does not closing a closed socket resulting in many CLOSE_WAIT]   This issue was patched in the following CDH releases: CDH5.13.0, CDH5.13.1, CDH5.13.2, CDH5.13.3, CDH5.14.0,  CDH5.14.2, CDH5.14.4, CDH5.15.0,  CDH5.15.1, CDH6.0.0.   [1] - upstream HBase JIRA - https://issues.apache.org/jira/browse/HBASE-9393?attachmentOrder=asc ... View more

Hi,     I am also facing similar issue can you tell me where I need to add hive-contib.jar , We are using cdh5.4.2. Please let me know the exact path. Thanks!! ... View more

Glad to hear! ... View more

@herdrick: no it's actually just fine to do it in the order Harsh mentioned.  CM will automatically deploy the configurations you specify on the Configuration tab of a service to that service's roles when you restart the service.  For example, if you modify a datanode specific property in CM, save the change and restart the service, then all the datanodes will get new copies of their hdfs-site.xml files upon startup.  The only reason to deploy the client configs that Harsh mentioned is for external client apps that want to utilize the cluster.  If you made changes that will affect the behavior of clients, then the client configs need to be re-deployed.  This can happen after the services are restarted, but before you attempt to reconnect to the cluster with your client app.  I hope that clears it up.   Clint ... View more