HiveServer2 has more than 10K waiting daemon threads named "Get-Input-Paths-"

[#1]

My HiveServer2 daemon thread "Get-Input-Paths-" count has been increased Continuously.

About a week later the HiveServer2 cpu reached 100%(single core) and all SQL could no be executed.

https://issues.apache.org/jira/browse/HIVE-16949

Environment:

CentOS Linux release 7.3.1611 (Core)  3.10.0-514.el7.x86_64

java version "1.8.0_121"

CDH & CM version: 5.11.1

thread dump file:

how to upload.........

this report is after HiveServer2 restart, but the thread still increase continously 

http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDcvMTUvLS1oaXZlc2VydmVyLnppcC0tMy0yNi0x...

snipt of the thread dump file:

"Get-Input-Paths-1" #37702 daemon prio=5 os_prio=0 tid=0x0000000001021000 nid=0x2ca15 waiting on condition [0x00007fb0046f0000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006de406738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None

"Get-Input-Paths-0" #37701 daemon prio=5 os_prio=0 tid=0x0000000001bb8800 nid=0x2ca14 waiting on condition [0x00007fafe99d8000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006de406738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None

oh~ I found the issue on JIRA

https://issues.apache.org/jira/browse/HIVE-16949 

[#2 · Accepted solution]

I think you are suffering from https://issues.apache.org/jira/browse/HIVE-16949.  I am suffering from it on my dev cluster.  Since I upgrade it CDH 5.11.1 the thread count has climbed steadily.  Just got reports today of HS2 slowing down on partition operations as well as throwing exceptions about concurrency issues.  I have added the workaround and will report on the status of it later.

The patch that introduced this leak, https://issues.apache.org/jira/browse/HIVE-15546, was added to CDH 5.11.0.