hivemetastore doing GC on hdp 3.1 with heap size 3GB and metastore processs terminates

[#1]

We are using hdp 3.1 We have give hive metastore heap 3GB of memory but it is doing GC and metastore process dies after that. the heapdump (hms_heapdump.hprof) showing the below in some class MutableQuantiles? Any pointers what could be going on wrong here.

[#2]

The image of heapdump not showing up so pasting the heapdump

Class Name | Shallow Heap | Retained Heap | Percentage

------------------------------------------------------------------------------------------------------------------------------------

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue @ 0x70128be78 | 32 | 2,997,787,240 | 97.85%

|- java.util.concurrent.RunnableScheduledFuture[898656] @ 0x7afe00000 | 3,594,640 | 2,997,787,184 | 97.85%

| |- java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask @ 0x79ba68588| 72 | 4,656 | 0.00%

| | '- java.util.concurrent.Executors$RunnableAdapter @ 0x79ba685d0 | 24 | 4,584 | 0.00%

| | '- org.apache.hadoop.metrics2.lib.MutableQuantiles$RolloverSample @ 0x79ba685e8| 16 | 4,560 | 0.00%

------------------------------------------------------------------------------------------------------------------------------------

[#3]

@bharat jain

Can you attach the HS2 logs, without the logs it's difficult to guess?

HTH

[#4]

My colleague Rajat has created a ticket for this at hadoop

https://issues.apache.org/jira/browse/HADOOP-16278

and this might be related to a similar issue reported at

https://issues.apache.org/jira/browse/HADOOP-16248