Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive Metastore Connection Failure then Retry

avatar
Expert Contributor

When running hive 1.2.1 on HDP 2.4 Hive successfully connects to the metastore and then later drops the connection. It seems like an exception is throwing after it successfully connects to metastore. I noticed if we turn off the CBO settings it will by pass the metastore and skip this exception. We are using ORC and have run compute stats.

2016-09-07 04:44:08,643 INFO [main]: hive.metastore (HiveMetaStoreClient.java:isCompatibleWith(296)) - Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook 2016-09-07 04:44:08,647 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(382)) - Trying to connect to metastore with URI INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(478)) - Connected to metastore.

--

2016-09-07 04:44:08,647 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(382)) - Trying to connect to metastore with URI 2016-09-07 04:44:08,649 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(478)) - Connected to metastore. 2016-09-07 04:44:08,664 INFO [main]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1173)) - mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 2016-09-07 04:44:08,729 WARN [main]: metastore.RetryingMetaStoreClient (RetryingMetaStoreClient.java:invoke(184)) - MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_aggr_stats_for(ThriftHiveMetastore.java:3033)

2016-09-07 04:44:13,784 WARN [main]: metastore.RetryingMetaStoreClient (RetryingMetaStoreClient.java:invoke(184)) - MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)

1 ACCEPTED SOLUTION

avatar
Cloudera Employee

@Kirk Haslbeck Currently there is open issue with aggregating date column statistics for partitioned tables.

https://issues.apache.org/jira/browse/HIVE-14773

When hive client queries the metastore db for date column statistics, it runs into a NPE. This may show up as increased lag time in Tez because it takes longer for DAG execution to kick in.

To workaround this, you can delete the column statistics for the data column from PART_COL_STATS in the metastore db.

View solution in original post

1 REPLY 1

avatar
Cloudera Employee

@Kirk Haslbeck Currently there is open issue with aggregating date column statistics for partitioned tables.

https://issues.apache.org/jira/browse/HIVE-14773

When hive client queries the metastore db for date column statistics, it runs into a NPE. This may show up as increased lag time in Tez because it takes longer for DAG execution to kick in.

To workaround this, you can delete the column statistics for the data column from PART_COL_STATS in the metastore db.