Hive Metastore NullPointerException get_partitions_by_expr

geko — Wed, 29 Jun 2016 17:41:15 GMT

Hello,

in an upgraded HDP cluster (upgrade to 2.3.4) the following ERROR is shown in Metastore log:

2016-06-29 08:32:40,184 ERROR [pool-5-thread-109]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(155)) -
MetaException(message:java.lang.NullPointerException)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5420)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rethrowException(HiveMetaStore.java:4542)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_expr(HiveMetaStore.java:4525)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
        at com.sun.proxy.$Proxy3.get_partitions_by_expr(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_expr.getResult(ThriftHiv
eMetastore.java:10218)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_partitions_by_expr.getResult(ThriftHiv
eMetastore.java:10202)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridg
e.java:681)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridg
e.java:676)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBri
dge.java:676)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.getExprString(ExprNodeGenericFuncDesc.java:158)
        at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.convertExprToFilter(PartitionExpressi
onForMetastore.java:40)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExprInternal(ObjectStore.java:2072)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByExpr(ObjectStore.java:2059)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
        at com.sun.proxy.$Proxy0.getPartitionsByExpr(Unknown Source)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_expr(HiveMetaStore.java:4520)
        ... 20 more

Any hint where to investigate ?

Many thanks

Re: Hive Metastore NullPointerException get_partitions_by_expr

jyadav — Wed, 29 Jun 2016 17:53:32 GMT

Looks like eException is matching with below bugs.

https://issues.apache.org/jira/browse/HIVE-10288

https://issues.apache.org/jira/browse/HIVE-8272

Re: Hive Metastore NullPointerException get_partitions_by_expr

geko — Wed, 29 Jun 2016 18:48:23 GMT

Hi @Jitendra Yadav , thanks for your reply.

Those issues are resolved and closed, and the Hive version which is in HDP 2.3.4 (Hive 1.2.1.2.3) should contain those fixes.

Any further hints how to investigate into this ?!?!

Re: Hive Metastore NullPointerException get_partitions_by_expr

geko — Wed, 29 Jun 2016 18:57:56 GMT

...and in addition, the linked issues are not corresponding to the get_partitions_by_expr function.

Re: Hive Metastore NullPointerException get_partitions_by_expr

pavel_razumtsev — Fri, 01 Jul 2016 20:39:23 GMT

Hi @Gerd Koenig

This is a bug in hive libraries of version 1.2.1.2.3.0.0 . Upgrade to 1.2.1.2.3.4.7-4 and it will work.

Re: Hive Metastore NullPointerException get_partitions_by_expr

geko — Sat, 02 Jul 2016 01:14:33 GMT

Hi @Pavel Razumtsev ,

thanks for the update.

question Re: Hive Metastore NullPointerException get_partitions_by_expr in Archives of Support Questions (Read Only)

Hive Metastore NullPointerException get_partitions_by_expr

Re: Hive Metastore NullPointerException get_partitions_by_expr

Re: Hive Metastore NullPointerException get_partitions_by_expr

Re: Hive Metastore NullPointerException get_partitions_by_expr

Re: Hive Metastore NullPointerException get_partitions_by_expr

Re: Hive Metastore NullPointerException get_partitions_by_expr