Hive Metastore Connection Failure then Retry

khaslbeck — Wed, 14 Sep 2016 20:33:24 GMT

When running hive 1.2.1 on HDP 2.4 Hive successfully connects to the metastore and then later drops the connection. It seems like an exception is throwing after it successfully connects to metastore. I noticed if we turn off the CBO settings it will by pass the metastore and skip this exception. We are using ORC and have run compute stats.

2016-09-07 04:44:08,643 INFO [main]: hive.metastore (HiveMetaStoreClient.java:isCompatibleWith(296)) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook 2016-09-07 04:44:08,647 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(382)) - Trying to connect to metastore with URI INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(478)) - Connected to metastore.

2016-09-07 04:44:08,647 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(382)) - Trying to connect to metastore with URI 2016-09-07 04:44:08,649 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(478)) - Connected to metastore. 2016-09-07 04:44:08,664 INFO [main]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1173)) - mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 2016-09-07 04:44:08,729 WARN [main]: metastore.RetryingMetaStoreClient (RetryingMetaStoreClient.java:invoke(184)) - MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_aggr_stats_for(ThriftHiveMetastore.java:3033)

2016-09-07 04:44:13,784 WARN [main]: metastore.RetryingMetaStoreClient (RetryingMetaStoreClient.java:invoke(184)) - MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)

Re: Hive Metastore Connection Failure then Retry

ndembla — Fri, 16 Sep 2016 20:08:13 GMT

@Kirk Haslbeck Currently there is open issue with aggregating date column statistics for partitioned tables.

https://issues.apache.org/jira/browse/HIVE-14773

When hive client queries the metastore db for date column statistics, it runs into a NPE. This may show up as increased lag time in Tez because it takes longer for DAG execution to kick in.

To workaround this, you can delete the column statistics for the data column from PART_COL_STATS in the metastore db.

question Hive Metastore Connection Failure then Retry in Archives of Support Questions (Read Only)

Hive Metastore Connection Failure then Retry

Re: Hive Metastore Connection Failure then Retry