Hive Metastore is getting down frequently

[#1]

SYMPTOM:

hive metastore process is going down frequently with following exceptions:

2016-12-16 01:30:45,016 ERROR [Thread[Thread-5,5,main]]: thrift.TokenStoreDelegationTokenSecretManager (TokenStoreDelegationTokenSecretManager.java:run(331)) - ExpiredTokenRemover thread received         unexpected exception. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: org.datanucleus.transaction.NucleusTransactionException: Invalid state. Transaction has already started
 org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: org.datanucleus.transaction.NucleusTransactionException: Invalid state. Transaction has already started
     at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:131)
     at org.apache.hadoop.hive.thrift.DBTokenStore.getToken(DBTokenStore.java:76)
     at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.removeExpiredTokens(TokenStoreDelegationTokenSecretManager.java:256)
     at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager$ExpiredTokenRemover.run(TokenStoreDelegationTokenSecretManager.java:319)
     at java.lang.Thread.run(Thread.java:745)
 Caused by: org.datanucleus.transaction.NucleusTransactionException: Invalid state. Transaction has already started
     at org.datanucleus.transaction.TransactionManager.begin(TransactionManager.java:47)
     at org.datanucleus.TransactionImpl.begin(TransactionImpl.java:131)
     at org.datanucleus.api.jdo.JDOTransaction.internalBegin(JDOTransaction.java:88)
     at org.datanucleus.api.jdo.JDOTransaction.begin(JDOTransaction.java:80)
     at org.apache.hadoop.hive.metastore.ObjectStore.openTransaction(ObjectStore.java:392)
     at org.apache.hadoop.hive.metastore.ObjectStore.getToken(ObjectStore.java:6412)
     at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
     at com.sun.proxy.$Proxy0.getToken(Unknown Source)
     at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:123)
     ... 4 more
 2016-12-16 01:30:45,017 INFO  [Thread-3]: metastore.HiveMetaStore (HiveMetaStore.java:run(5643)) - Shutting down hive metastore.

ROOT CAUSE:

when hive metastore start, the DelegationTokenSecretManager will maintain the same objectstore,this lead to the concurrent issue.

WORKAROUND:

NA

RESOLUTION:

this concurrency issue has been fixed in HIVE-11616 and HIVE-13090 so need to apply the patches.