Reply
Explorer
Posts: 23
Registered: ‎01-11-2018

HiveServer2 growing number of file descriptors - connection leak

Hi!

 

On our CDH 5.9.3 cluster we experience problem with growing number of HiveServer2, what forces us to restart instances of the HiveServer every 2 weeks or so. In our case most of the open file descriptors are actually TCP connections to lots Data Nodes on port 1004. Has anybody else had similar problem before? Is there any fix available for this issue? Can it be related to any of these two bugs:

https://issues.apache.org/jira/browse/HIVE-1185

https://issues.apache.org/jira/browse/HIVE-7081

 

Thanks for any help.

 

Master
Posts: 326
Registered: ‎07-01-2015

Re: HiveServer2 growing number of file descriptors - connection leak

I had from time to time a significant increase of file descriptors (up to 20k) but it was usually because of high concurrency or very large spark jobs. The descriptors were released when the jobs finished. I also noticed that during that peak I had almost 10k open operations. (CDH5.13)
Explorer
Posts: 23
Registered: ‎01-11-2018

Re: HiveServer2 growing number of file descriptors - connection leak

@Tomas79Thank you very much for your reply. We do run spark apps on our cluster but in our case we don't see large number of open operations - both 'Open connections' and 'Open operations' are on relatively low level, below 30. I also noticed that probably I recognized the problem as 'leak' prematurely, as when cluster utilization is low, the number of open descriptors is also much lower, what wouldn't happen if connections hanged in an incorrect state. Now I'm coming to conclusion that this situation is a consequence of high Hive utilization and many small files around HDFS. Thank you for your time, Cheers

Champion
Posts: 746
Registered: ‎05-16-2016

Re: HiveServer2 growing number of file descriptors - connection leak

is your HS2 going down ? or it just show bad health ?
Explorer
Posts: 23
Registered: ‎01-11-2018

Re: HiveServer2 growing number of file descriptors - connection leak

@csgunaNo, actually we restart HiveServer in controlled conditions when it reaches 25k+ open file descriptors. Now we plan to raise the number of allowed file descriptors for HiveServer.

Cloudera Employee
Posts: 425
Registered: ‎03-23-2015

Re: HiveServer2 growing number of file descriptors - connection leak

Can you please check do you have session timeout for HiveServer2? If no session timeout, all sessions will keep open endlessly and files will be kept open for a long time.

Worth a check.
Highlighted
Explorer
Posts: 23
Registered: ‎01-11-2018

Re: HiveServer2 growing number of file descriptors - connection leak

Hi, actually both session and operation timeouts are set  to 6h, so this shouldn't be a problem. Thanks!

Announcements