Support Questions

Find answers, ask questions, and share your expertise

CDH5.11.0 have bug in hiveserver2

New Contributor
Hi,Now i'm use 5.11.0release,when i use hiveserver2 ,there are some question, the hive i used 8days ago,the job of use hiveserver2 become too slow, and spent too time .I find the reason, the hiverserver2‘s have much thread are in blocked status, the sum of blocked status is 12000, the next is the memory of hiveserver2 still too high.when i restart the hiveserver2 ,all of then become normal., the old version(5.7.1) don't hive this problem ,Maybe from 5.11.0 have input the bug ,cause hiveserver2 become too slow,Who can help solve the problem ,thanks !
5 REPLIES 5

Champion
There was a bug added, I thought it was added in CDH 5.11.1, but I may be off on that.

It was added when the community added parallelism to the input path listing process. Threads are leaked and after some time this will slow down HS2. There is a workaround that I have confirmed works. Please review the below post and subsequent JIRA.

https://community.cloudera.com/t5/Batch-SQL-Apache-Hive/HiveServer2-has-more-than-10K-waiting-daemon...

New Contributor
No,The Problem Is not the same as you said, when I update 5.11.0 to 5.11.1,the problem still ,and when restart the hiveserver2 deamon,all of they become normal.but , 7days later, use  jstack -l pid | grep BLOCKED |wc -l ,the sum of this also hive 10k or more.all of the sql job become slow ,especially create partition 

Champion
The bug I mentioned is in CDH 5.11.1 and hasn't been fixed yet. The workaround is to change the number of threads available to the input listing process. Change this value and restart HS2. You should stop seeing such high number of threads.

hive.exec.input.listing.max.threads = 1

New Contributor

I am using CDH 5.12.1 but facing the same issue.

Champion

Can you share graphs showing the thread leak?

 

Also, I don't see this patch for the bug in any CDH release to date.  So you will need to apply the workaround listed in the patch notes or in the other community thread.

 

https://issues.apache.org/jira/browse/HIVE-16949