CDH 4.3 MetaStoreClient lost connection. Attempting to reconnect

[#1]

19:56:40.570	INFO	org.apache.hadoop.hive.ql.ppd.OpProcFactory	(pers_id_type) IN ('lol', 'bob', 'tom')
19:57:00.597	WARN	org.apache.hadoop.hive.metastore.RetryingMetaStoreClient	MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_with_auth(ThriftHiveMetastore.java:1391) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_with_auth(ThriftHiveMetastore.java:1374) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsWithAuthInfo(HiveMetaStoreClient.java:692) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74) at $Proxy9.listPartitionsWithAuthInfo(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.getPartitions(Hive.java:1565) at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:202) at org.apache.hadoop.hive.ql.optimizer.pcr.PcrOpProcFactory$FilterPCR.process(PcrOpProcFactory.java:112) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:87) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:124) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:101) at org.apache.hadoop.hive.ql.optimizer.pcr.PartitionConditionRemover.transform(PartitionConditionRemover.java:86) at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:102) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8200) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:355) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:95) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:76) at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:114) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:194) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:154) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:190) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1193) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1178) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 43 more
19:57:01.598	INFO	hive.metastore	Trying to connect to metastore with URI thrift://prod-beeswax.lol.ru:9083
19:57:01.600	INFO	hive.metastore	Waiting 1 seconds before next connection attempt.
19:57:02.600	INFO	hive.metastore	Connected to metastore.
19:57:22.621	ERROR	hive.ql.metadata.Hive	org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_with_auth(ThriftHiveMetastore.java:1391)

[#2]

Omg. who had set limit for message length?

When normal HTML support would be? Google groups does handle Cloudera logs better.

We are using CDH 4.3, I don't underastand what we are doing wrong... It happens on several tables, the other tables are fine

[#3]

Sergey,

From the looks of it, it seems like the hive-metastore service is not running. I am assuming you are running a remote metastore. Can you make sure it's up and running? Perhaps, telnet to the machine it's meant to running on, on port 9083 to see if all the services are up, listening and the ports are open.

[#4]

We use CDH 4.3, CM 4.6.3

The first thing we've checked was metastore service. It works.

impala queries, other hive queries run fine. All our "SQL-like" stuff uses single metastore in PostGre DB. I've also see how does hive tried to get partitions for table used in failed query. There no exceptions on Metastore side, it works 100%

We've mentioned one more intersting thing.

Queried table have ~1500 partitions (hour partitions). When we issue the same query explicitly specifing the range partition, it works. Even if we specify the range between the oldest and the newest partition (the same as we don't sepcify them at all!) it works.

I have no Idea what is that...