Hive Metastore becomes unresponsive

[#1]

ISSUE : Hive Metastore stops responding until a restart happens.

WORKAROUND :

Please collect the jstack output 5 times at an interval of 30 seconds when hivemetastore is hung.

jstack -F <hivemetastorepid> > jstack.out

Then analyze the jstack output and look for the keyword "locked".

If you see something similar to below then there are locked threads with respect to filesystem cache.

"pool-5-thread-180" #12021 prio=5 os_prio=0 tid=0x00007f63de37e000 nid=0x3900 runnable [0x00007f6391fc3000]
   java.lang.Thread.State: RUNNABLE
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886)
- locked <0x00000005c0032ca8> (a org.apache.hadoop.fs.FileSystem$Cache)
at org.apache.hadoop.fs.FileSystem.closeAllForUGI(FileSystem.java:473)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:131)

Please add the following to hive-site.xml and restart Hive services. These properties will disable the hdfs filesystem cache.

fs.hdfs.impl.disable.cache=true

fs.file.impl.disable.cache=true