Hive Metastore fails to start after upgrading to CM 5.14.1 from CM 5.12.2

[#1]

Hi,
After upgrading from CM 5.12.2 to CM 5.14.1 the Hive metastore service can't start and give the below error message:

Command failed to run because this role has an invalid configuration. Review and correct its configuration. First error: Enable Stored Notifications in Database is required by Sentry and must be enabled.

And when I enable the Enable Stored Notifications in Database, I have this error:

2018-03-14 19:24:27,154 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [main]: Starting hive metastore on port 9083
2018-03-14 19:24:27,230 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.sentry.conf.url does not exist
2018-03-14 19:24:27,468 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [main]: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2018-03-14 19:24:27,607 INFO  org.apache.hadoop.hive.metastore.ObjectStore: [main]: ObjectStore, initialize called
2018-03-14 19:24:27,737 INFO  DataNucleus.Persistence: [main]: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
2018-03-14 19:24:27,737 INFO  DataNucleus.Persistence: [main]: Property datanucleus.cache.level2 unknown - will be ignored
2018-03-14 19:24:28,209 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.sentry.conf.url does not exist
2018-03-14 19:24:28,211 INFO  org.apache.hadoop.hive.metastore.ObjectStore: [main]: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2018-03-14 19:24:28,257 INFO  org.apache.hadoop.hive.metastore.ObjectStore: [main]: Initialized ObjectStore
2018-03-14 19:24:29,217 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [main]: Added admin role in metastore
2018-03-14 19:24:29,219 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [main]: Added public role in metastore
2018-03-14 19:24:29,266 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [main]: No user is added in admin role, since config is empty
2018-03-14 19:24:29,266 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [main]: Begin calculating metadata count metrics.
2018-03-14 19:24:29,268 INFO  DataNucleus.Datastore: [main]: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
2018-03-14 19:24:29,269 INFO  DataNucleus.Datastore: [main]: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
2018-03-14 19:24:29,332 INFO  DataNucleus.Datastore: [main]: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
2018-03-14 19:24:29,332 INFO  DataNucleus.Datastore: [main]: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
2018-03-14 19:24:29,364 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [main]: Finished metadata count metrics: 1 databases, 3 tables, 0 partitions.
2018-03-14 19:24:29,385 INFO  org.apache.sentry.binding.hive.conf.HiveAuthzConf: [main]: DefaultFS: hdfs://node.master.dc8:8020
2018-03-14 19:24:29,400 INFO  org.apache.sentry.binding.hive.conf.HiveAuthzConf: [main]: DefaultFS: hdfs://node.master.dc8:8020
2018-03-14 19:24:29,438 WARN  org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.sentry.conf.url does not exist
2018-03-14 19:24:29,439 WARN  org.apache.sentry.binding.hive.conf.HiveAuthzConf: [main]: Using the deprecated config setting hive.sentry.server instead of sentry.hive.server
2018-03-14 19:24:29,442 ERROR org.apache.hadoop.hive.metastore.HiveMetaStore: [main]: MetaException(message:Failed to instantiate listener named: org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener, reason: java.lang.ClassNotFoundException: org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getMetaStoreListeners(MetaStoreUtils.java:1517)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:555)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6313)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6308)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6558)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6485)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

2018-03-14 19:24:29,442 ERROR org.apache.hadoop.hive.metastore.HiveMetaStore: [main]: Metastore Thrift Server threw an exception...
MetaException(message:Failed to instantiate listener named: org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener, reason: java.lang.ClassNotFoundException: org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getMetaStoreListeners(MetaStoreUtils.java:1517)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:555)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6313)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6308)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6558)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6485)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
2018-03-14 19:24:29,444 INFO  org.apache.hadoop.hive.metastore.HiveMetaStore: [Thread-2]: Shutting down hive metastore.

NB: Knowing that CDH is always 5.12.2 and JDK updated from 1.8_121 to 1.8_144.

Is it just a temporary problem that need a CDH be upgraded to 5.14.0 ? or it's a bug!?

Thanks in advance.

[#2 · Accepted solution]

Hi,

The problem was that the class : org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener don't exist in the jar file /opt/cloudera/parcels/CDH-5.12.2-1.cdh5.12.2.p0.4/jars/sentry-binding-hive-1.5.1-cdh5.12.2.jar within the CDH 5.12.2.

In fact, there is two ways to solve it, either upgrade to CDH-5.14.0 (this is the easy way) or download the sentry tarball sentry-1.5.1-cdh5.14.0.tar.gz, extract it and try to replace the exist jar with sentry-1.5.1-cdh5.14.0/apache-sentry-1.5.1-cdh5.14.0-bin/lib/sentry-binding-hive-1.5.1-cdh5.14.0.jar that contain the concerned class org/apache/sentry/binding/metastore/SentrySyncHMSNotificationsPostEventListener.class.

[#3]

Hi,

This looks like a different flavor of something we encountered while testing Sentry in 5.13 (that we didn't resolve). I pinged a few folks internally.

FWIW I tried just spinning up a secure CM5.14.1 + CDH5.12.2 cluster with all the services and my HMS came up just fine without that config. Can you describe more how you upgraded CM, if any other configs were changed before/after the upgrade, and if you added any services before or after?

As an aside, the Impala forum is probably not the best place to get help for this issue.

Thanks,

J-D