Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive unable to stabilize connection to metastore in local mode on mysql

Hive unable to stabilize connection to metastore in local mode on mysql

New Contributor

Hi esteemed Hadoop contributors,

I am new to this world and I have been trying to run the simple twitter example using Cloudera CDH 5.1.0-1.cdh5.1.0.p0.53

with the exact instructions provided on github for twitter example.I have tried to configure metastore on mysql in local mode.

 

After trying multiple times, I am unable to get a stable Metastore connection. The connection seems to drop.

 

I haverun Hive in debug mode and am getting following information when "show tables" command is run on Hive command prompt

14/08/09 11:55:56 INFO hive.metastore: Connected to metastore.
14/08/09 11:56:16 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 11:56:36 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 11:56:56 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 11:57:16 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 11:57:36 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 11:57:56 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 11:58:16 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 0ms
14/08/09 11:58:36 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 11:58:56 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 11:58:56 WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:444)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:431)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:854)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at com.sun.proxy.$Proxy11.getDatabase(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1142)
at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1131)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2189)

at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1263)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:921)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:790)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 36 more

14/08/09 11:58:57 INFO hive.metastore: Trying to connect to metastore with URI thrift://localhost.localdomain:9083
14/08/09 11:58:57 DEBUG security.Groups: Returning fetched groups for 'root'
14/08/09 11:59:16 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 11:59:36 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 2ms
14/08/09 11:59:56 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 12:00:16 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 12:00:36 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 12:00:56 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 12:01:16 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 12:01:36 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 12:01:56 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 12:01:57 WARN hive.metastore: set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:2855)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:2841)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:312)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:205)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:87)
at com.sun.proxy.$Proxy11.getDatabase(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1142)
at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1131)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2189)

at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1263)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:921)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:790)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 33 more
14/08/09 12:01:57 INFO hive.metastore: Connected to metastore.
14/08/09 12:02:16 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms

14/8/09 12:03:16 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 0ms
14/08/09 12:03:36 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 0ms
14/08/09 12:03:56 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 12:04:16 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 12:04:36 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 0ms
14/08/09 12:04:56 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x147b94d53f80055 after 1ms
14/08/09 12:04:57 ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1146)
at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1131)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2189)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1263)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)

at org.apache.hadoop.hive.ql.Driver.run(Driver.java:921)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:790)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_database(ThriftHiveMetastore.java:444)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:431)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:854)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at com.sun.proxy.$Proxy11.getDatabase(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1142)
... 21 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 36 more

14/08/09 12:04:57 INFO log.PerfLogger: </PERFLOG method=task.DDL.Stage-0 start=1407565376091 end=1407566097858 duration=721767 from=org.apache.hadoop.hive.ql.Driver>
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out

14/08/09 12:04:57 INFO log.PerfLogger: </PERFLOG method=task.DDL.Stage-0 start=1407565376091 end=1407566097858 duration=721767 from=org.apache.hadoop.hive.ql.Driver>
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
14/08/09 12:04:57 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
14/08/09 12:04:57 INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1407565376086 end=1407566097859 duration=721773 from=org.apache.hadoop.hive.ql.Driver>
14/08/09 12:04:57 INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
14/08/09 12:04:57 INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1407566097859 end=1407566097859 duration=0 from=org.apache.hadoop.hive.ql.Driver>
14/08/09 12:04:57 INFO exec.ListSinkOperator: 0 finished. closing...
14/08/09 12:04:57 INFO exec.ListSinkOperator: 0 forwarded 0 rows
14/08/09 12:04:58 INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
14/08/09 12:04:58 INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1407566098005 end=1407566098006 duration=1 from=org.apache.hadoop.hive.ql.Driver>
14/08/09 12:04:58 DEBUG zookeeper.ZooKeeper: Closing session: 0x147b94d53f80055
14/08/09 12:04:58 DEBUG zookeeper.ClientCnxn: Closing client for session: 0x147b94d53f80055
14/08/09 12:04:58 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x147b94d53f80055, packet:: clientPath:null serverPath:null finished:false header:: 2,-11 replyHeader:: 2,16873,0 request:: null response:: null
14/08/09 12:04:58 DEBUG zookeeper.ClientCnxn: An exception was thrown while closing send thread for session 0x147b94d53f80055 : Unable to read additional data from server sessionid 0x147b94d53f80055, likely server has closed socket
14/08/09 12:04:58 DEBUG zookeeper.ClientCnxn: Disconnecting client for session: 0x147b94d53f80055
14/08/09 12:04:58 INFO zookeeper.ClientCnxn: EventThread shut down
14/08/09 12:04:58 INFO zookeeper.ZooKeeper: Session: 0x147b94d53f80055 closed

 

Kindly help me in resolving this issue.

 

Thanks and Regards,

Ketan

2 REPLIES 2

Re: Hive unable to stabilize connection to metastore in local mode on mysql

New Contributor

 Hi Dear contributors,

Never mind. I found the answer after lot of reconfigurations and some logical analysis of Console entries when hive was rin in debug mode.

 

Thanks,

Ketan

Re: Hive unable to stabilize connection to metastore in local mode on mysql

Cloudera Employee
could you share the configuration changes to solve this set_gui issue and stabilize the HMS? Thanks