HiveServer2 growing number of file descriptors - connection leak

[#1]

Hi!

On our CDH 5.9.3 cluster we experience problem with growing number of HiveServer2, what forces us to restart instances of the HiveServer every 2 weeks or so. In our case most of the open file descriptors are actually TCP connections to lots Data Nodes on port 1004. Has anybody else had similar problem before? Is there any fix available for this issue? Can it be related to any of these two bugs:

https://issues.apache.org/jira/browse/HIVE-1185

https://issues.apache.org/jira/browse/HIVE-7081

Thanks for any help.

[#2]

I had from time to time a significant increase of file descriptors (up to 20k) but it was usually because of high concurrency or very large spark jobs. The descriptors were released when the jobs finished. I also noticed that during that peak I had almost 10k open operations. (CDH5.13)

[#3]

@Tomas79Thank you very much for your reply. We do run spark apps on our cluster but in our case we don't see large number of open operations - both 'Open connections' and 'Open operations' are on relatively low level, below 30. I also noticed that probably I recognized the problem as 'leak' prematurely, as when cluster utilization is low, the number of open descriptors is also much lower, what wouldn't happen if connections hanged in an incorrect state. Now I'm coming to conclusion that this situation is a consequence of high Hive utilization and many small files around HDFS. Thank you for your time, Cheers

[#4]

is your HS2 going down ? or it just show bad health ?

[#5]

@csgunaNo, actually we restart HiveServer in controlled conditions when it reaches 25k+ open file descriptors. Now we plan to raise the number of allowed file descriptors for HiveServer.

[#6]

Can you please check do you have session timeout for HiveServer2? If no session timeout, all sessions will keep open endlessly and files will be kept open for a long time.

Worth a check.

[#7]

Hi, actually both session and operation timeouts are set  to 6h, so this shouldn't be a problem. Thanks!