10-18-2018 07:24 AM
On our CDH 5.9.3 cluster we experience problem with growing number of HiveServer2, what forces us to restart instances of the HiveServer every 2 weeks or so. In our case most of the open file descriptors are actually TCP connections to lots Data Nodes on port 1004. Has anybody else had similar problem before? Is there any fix available for this issue? Can it be related to any of these two bugs:
Thanks for any help.
10-19-2018 03:16 PM
10-21-2018 09:12 AM
@Tomas79Thank you very much for your reply. We do run spark apps on our cluster but in our case we don't see large number of open operations - both 'Open connections' and 'Open operations' are on relatively low level, below 30. I also noticed that probably I recognized the problem as 'leak' prematurely, as when cluster utilization is low, the number of open descriptors is also much lower, what wouldn't happen if connections hanged in an incorrect state. Now I'm coming to conclusion that this situation is a consequence of high Hive utilization and many small files around HDFS. Thank you for your time, Cheers
10-23-2018 10:11 AM
@csgunaNo, actually we restart HiveServer in controlled conditions when it reaches 25k+ open file descriptors. Now we plan to raise the number of allowed file descriptors for HiveServer.
10-25-2018 04:15 AM