Reply
Highlighted
Posts: 6
Topics: 1
Kudos: 1
Solutions: 1
Registered: ‎06-07-2016

Hive Metastore RPC fails on Sentry HDFS MetastorePlugin after upgrade to CDH 5.8.2 on Oracle BDA

We have recently upgraded an Oracle BDA from CDH 5.8.0 to CDH 5.8.2, after the upgrade we encounter an error with Hive Metastore RPC when we run a DDL command.

 

Every time we run a DDL command the command fails with an error with a NullPointerPointer.

Anyway the command was be executed successfully, in Hive, Beeline, Impala-Shell and Hue we can work as the error was not raised.

Unfortunately some tools, like Sqoop, stop to work after the error.

 

Looking into HiveMetastore Log we find this entry:

2016-11-14 12:43:50,043 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: [pool-6-thread-4]: MetaException(message:java.lang.NullPointerException)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5600)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:967)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:138)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
        at com.sun.proxy.$Proxy8.create_database(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8863)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:8847)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:735)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:730)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:730)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
        at org.apache.sentry.hdfs.MetastorePlugin.applyLocal(MetastorePlugin.java:344)
        at org.apache.sentry.hdfs.MetastorePlugin.processUpdate(MetastorePlugin.java:375)
        at org.apache.sentry.hdfs.MetastorePlugin.notifySentryAndApplyLocal(MetastorePlugin.java:352)
        at org.apache.sentry.hdfs.MetastorePlugin.addPath(MetastorePlugin.java:229)
        at org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.onCreateDatabase(SentryMetastorePostEventListener.java:163)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:927)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:956)
        ... 20 more

Where we found that the Sentry Plugin raises the error. 

 

Il looks the same error as this sentry issue: [SENTRY-1270] Improve error handling - Database with malformed URI causes NPE in HMS plugin during D....

 

To be honest I do not know if is an Oracle BDA issue or a CDH issue.