I think you are suffering from https://issues.apache.org/jira/browse/HIVE-16949.  I am suffering from it on my dev cluster.  Since I upgrade it CDH 5.11.1 the thread count has climbed steadily.  Just got reports today of HS2 slowing down on partition operations as well as throwing exceptions about concurrency issues.  I have added the workaround and will report on the status of it later.   The patch that introduced this leak, https://issues.apache.org/jira/browse/HIVE-15546, was added to CDH 5.11.0. ... View more

Yes, been available for a while, but it's a separate parallel install so as to not replace Spark 1.x https://www.cloudera.com/downloads/spark2/2-1.html ... View more

I restarted the cm-agent on the error server. The problem solved.. ... View more

solved above ... View more

  Now the cluster that previously didn't see cdh 5.4.1 parcels got 5.4.2-1.cdh5.4.2.p0.2 parcel update available. I believe this is probably the cause. (cloudera defer the 5.4.1 and waited for the 5.4.2 being pushed instead, because the release time is quite close)   Look: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_rn_new_in_542.html ... View more