Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HiveServer2 growing number of file descriptors - connection leak

HiveServer2 growing number of file descriptors - connection leak

Contributor

Hi!

 

On our CDH 5.9.3 cluster we experience problem with growing number of HiveServer2, what forces us to restart instances of the HiveServer every 2 weeks or so. In our case most of the open file descriptors are actually TCP connections to lots Data Nodes on port 1004. Has anybody else had similar problem before? Is there any fix available for this issue? Can it be related to any of these two bugs:

https://issues.apache.org/jira/browse/HIVE-1185

https://issues.apache.org/jira/browse/HIVE-7081

 

Thanks for any help.

 

6 REPLIES 6

Re: HiveServer2 growing number of file descriptors - connection leak

Master Collaborator
I had from time to time a significant increase of file descriptors (up to 20k) but it was usually because of high concurrency or very large spark jobs. The descriptors were released when the jobs finished. I also noticed that during that peak I had almost 10k open operations. (CDH5.13)

Re: HiveServer2 growing number of file descriptors - connection leak

Contributor

@Tomas79Thank you very much for your reply. We do run spark apps on our cluster but in our case we don't see large number of open operations - both 'Open connections' and 'Open operations' are on relatively low level, below 30. I also noticed that probably I recognized the problem as 'leak' prematurely, as when cluster utilization is low, the number of open descriptors is also much lower, what wouldn't happen if connections hanged in an incorrect state. Now I'm coming to conclusion that this situation is a consequence of high Hive utilization and many small files around HDFS. Thank you for your time, Cheers

Re: HiveServer2 growing number of file descriptors - connection leak

Champion
is your HS2 going down ? or it just show bad health ?
Highlighted

Re: HiveServer2 growing number of file descriptors - connection leak

Contributor

@csgunaNo, actually we restart HiveServer in controlled conditions when it reaches 25k+ open file descriptors. Now we plan to raise the number of allowed file descriptors for HiveServer.

Re: HiveServer2 growing number of file descriptors - connection leak

Guru
Can you please check do you have session timeout for HiveServer2? If no session timeout, all sessions will keep open endlessly and files will be kept open for a long time.

Worth a check.

Re: HiveServer2 growing number of file descriptors - connection leak

Contributor

Hi, actually both session and operation timeouts are set  to 6h, so this shouldn't be a problem. Thanks!