Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HiveServer2 has more than 10K waiting daemon threads named "Get-Input-Paths-"

SOLVED Go to solution

HiveServer2 has more than 10K waiting daemon threads named "Get-Input-Paths-"

Contributor

My HiveServer2 daemon thread "Get-Input-Paths-" count has been increased Continuously.

About a week later the HiveServer2 cpu reached 100%(single core) and all SQL could no be executed.

 

 

https://issues.apache.org/jira/browse/HIVE-16949

 

Environment:

CentOS Linux release 7.3.1611 (Core)  3.10.0-514.el7.x86_64

java version "1.8.0_121"

CDH & CM version: 5.11.1

 

123.png

 

thread dump file:

how to upload.........

 

this report is after HiveServer2 restart, but the thread still increase continously 

 

http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDcvMTUvLS1oaXZlc2VydmVyLnppcC0tMy0yNi0x...

 

snipt of the thread dump file:

"Get-Input-Paths-1" #37702 daemon prio=5 os_prio=0 tid=0x0000000001021000 nid=0x2ca15 waiting on condition [0x00007fb0046f0000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006de406738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None

"Get-Input-Paths-0" #37701 daemon prio=5 os_prio=0 tid=0x0000000001bb8800 nid=0x2ca14 waiting on condition [0x00007fafe99d8000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006de406738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- None

 

 

oh~ I found the issue on JIRA

https://issues.apache.org/jira/browse/HIVE-16949 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HiveServer2 has more than 10K waiting daemon threads named "Get-Input-Paths-"

Champion

I think you are suffering from https://issues.apache.org/jira/browse/HIVE-16949.  I am suffering from it on my dev cluster.  Since I upgrade it CDH 5.11.1 the thread count has climbed steadily.  Just got reports today of HS2 slowing down on partition operations as well as throwing exceptions about concurrency issues.  I have added the workaround and will report on the status of it later.

 

The patch that introduced this leak, https://issues.apache.org/jira/browse/HIVE-15546, was added to CDH 5.11.0.

1 REPLY 1

Re: HiveServer2 has more than 10K waiting daemon threads named "Get-Input-Paths-"

Champion

I think you are suffering from https://issues.apache.org/jira/browse/HIVE-16949.  I am suffering from it on my dev cluster.  Since I upgrade it CDH 5.11.1 the thread count has climbed steadily.  Just got reports today of HS2 slowing down on partition operations as well as throwing exceptions about concurrency issues.  I have added the workaround and will report on the status of it later.

 

The patch that introduced this leak, https://issues.apache.org/jira/browse/HIVE-15546, was added to CDH 5.11.0.