Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HiveServer2 growing number of file descriptors - connection leak

avatar
Contributor

Hi!

 

On our CDH 5.9.3 cluster we experience problem with growing number of HiveServer2, what forces us to restart instances of the HiveServer every 2 weeks or so. In our case most of the open file descriptors are actually TCP connections to lots Data Nodes on port 1004. Has anybody else had similar problem before? Is there any fix available for this issue? Can it be related to any of these two bugs:

https://issues.apache.org/jira/browse/HIVE-1185

https://issues.apache.org/jira/browse/HIVE-7081

 

Thanks for any help.

 

6 REPLIES 6

avatar
I had from time to time a significant increase of file descriptors (up to 20k) but it was usually because of high concurrency or very large spark jobs. The descriptors were released when the jobs finished. I also noticed that during that peak I had almost 10k open operations. (CDH5.13)

avatar
Contributor

@Tomas79Thank you very much for your reply. We do run spark apps on our cluster but in our case we don't see large number of open operations - both 'Open connections' and 'Open operations' are on relatively low level, below 30. I also noticed that probably I recognized the problem as 'leak' prematurely, as when cluster utilization is low, the number of open descriptors is also much lower, what wouldn't happen if connections hanged in an incorrect state. Now I'm coming to conclusion that this situation is a consequence of high Hive utilization and many small files around HDFS. Thank you for your time, Cheers

avatar
Champion
is your HS2 going down ? or it just show bad health ?

avatar
Contributor

@csgunaNo, actually we restart HiveServer in controlled conditions when it reaches 25k+ open file descriptors. Now we plan to raise the number of allowed file descriptors for HiveServer.

avatar
Super Guru
Can you please check do you have session timeout for HiveServer2? If no session timeout, all sessions will keep open endlessly and files will be kept open for a long time.

Worth a check.

avatar
Contributor

Hi, actually both session and operation timeouts are set  to 6h, so this shouldn't be a problem. Thanks!