Reply
Explorer
Posts: 64
Registered: ‎08-07-2017

hive metastore canary health bad in cloudera cluster

[ Edited ]

Hi,

 

Hive metastore canary health is bad in the cloudera cluster. I started the httpd service and checked the logs and I can see error messages as below:

 

Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:698)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:685)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:958)
	at org.apache.hive.service.cli.CLIService.start(CLIService.java:149)
	at org.apache.hive.service.CompositeService.start(CompositeService.java:70)
	at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:264)
	at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:310)
	at org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:68)
	at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:523)
	at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:396)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
	... 21 more
Converting exception to MetaException

Error starting services HiveServer2
org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
	at org.apache.hive.service.cli.CLIService.start(CLIService.java:151)
	at org.apache.hive.service.CompositeService.start(CompositeService.java:70)
	at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:264)
	at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:310)
	at org.apache.hive.service.server.HiveServer2.access$400(HiveServer2.java:68)
	at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:523)
	at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:396)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1178)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:960)
	at org.apache.hive.service.cli.CLIService.start(CLIService.java:149)
	... 12 more

Can you please help me with this?


 

Explorer
Posts: 64
Registered: ‎08-07-2017

Re: hive metastore canary health bad in cloudera cluster

Hi,

 

I am unable to access hive metastore database from the hive metastore server host in mysql shell with the hive user id and password.

 

I also don’t have the password for hive account  and  password in hive-site.xml not working.

 

I can see below error in the logs.I tried restarting the hive service , it is successfully starting and after 2 minutes metastore canary is breaking.

 

Failed initialising database.

Unable to open a test connection to the given database. JDBC url = jdbc:mysql://linuxUL.ad.infosys.com:3306/hive?useUnicode=true&characterEncoding=UTF-8, username = hive. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------

java.sql.SQLException: Access denied for user 'hive'@'linux170.ad.infosys.com' (using password: YES)

 

Please suggest.

 

Thanks,

Priya

Expert Contributor
Posts: 152
Registered: ‎07-01-2015

Re: hive metastore canary health bad in cloudera cluster

If you cannot connect to a mysql from command line with the user and password, why you expect to connect the metastore?
Try to nail down the problem in command line, install mysql client tools, try to log in with the hive user and password and if it works, then go back to CM and check if the user/pwd is correct.
Explorer
Posts: 64
Registered: ‎08-07-2017

Re: hive metastore canary health bad in cloudera cluster

I did reset of password and granted privilges and restarted the hive service.
It is working now.

Thanks,
Announcements