Reply
Expert Contributor
Posts: 162
Registered: ‎07-29-2013

CDH 4.3 MetaStoreClient lost connection. Attempting to reconnect

19:56:40.570INFOorg.apache.hadoop.hive.ql.ppd.OpProcFactory
(pers_id_type) IN ('lol', 'bob', 'tom')
19:57:00.597WARNorg.apache.hadoop.hive.metastore.RetryingMetaStoreClient
MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_with_auth(ThriftHiveMetastore.java:1391)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_with_auth(ThriftHiveMetastore.java:1374)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsWithAuthInfo(HiveMetaStoreClient.java:692)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74)
at $Proxy9.listPartitionsWithAuthInfo(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartitions(Hive.java:1565)
at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:202)
at org.apache.hadoop.hive.ql.optimizer.pcr.PcrOpProcFactory$FilterPCR.process(PcrOpProcFactory.java:112)
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:87)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:124)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:101)
at org.apache.hadoop.hive.ql.optimizer.pcr.PartitionConditionRemover.transform(PartitionConditionRemover.java:86)
at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:102)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8200)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:355)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:95)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:76)
at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:114)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:194)
at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:154)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:190)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1193)
at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1178)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 43 more
19:57:01.598INFOhive.metastore
Trying to connect to metastore with URI thrift://prod-beeswax.lol.ru:9083
19:57:01.600INFOhive.metastore
Waiting 1 seconds before next connection attempt.
19:57:02.600INFOhive.metastore
Connected to metastore.
19:57:22.621ERRORhive.ql.metadata.Hive
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_with_auth(ThriftHiveMetastore.java:1391)

Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: CDH 4.3 MetaStoreClient lost connection. Attempting to reconnect

Omg. who had set limit for message length?

When normal HTML support would be? Google groups does handle Cloudera logs better.

 

We are using CDH 4.3, I don't underastand what we are doing wrong... It happens on several tables, the other tables are fine

Cloudera Employee
Posts: 62
Registered: ‎07-29-2013

Re: CDH 4.3 MetaStoreClient lost connection. Attempting to reconnect

Sergey,

From the looks of it, it seems like the hive-metastore service is not running. I am assuming you are running a remote metastore. Can you make sure it's up and running? Perhaps, telnet to the machine it's meant to running on, on port 9083 to see if all the services are up, listening and the ports are open.

Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: CDH 4.3 MetaStoreClient lost connection. Attempting to reconnect

We use CDH 4.3, CM 4.6.3

The first thing we've checked was metastore service. It works.

impala queries, other hive queries run fine. All our "SQL-like" stuff uses single metastore in PostGre DB. I've also see how does hive tried to get partitions for table used in failed query. There no exceptions on Metastore side, it works 100%

 

We've mentioned one more intersting thing.

Queried table have ~1500 partitions (hour partitions). When we issue the same query explicitly specifing the range partition, it works. Even if we specify the range between the oldest and the newest partition (the same as we don't sepcify them at all!) it works.

 

I have no Idea what is that...