Hive Metastore fails to start properly after reboot

ScottChris — Fri, 16 Sep 2022 10:13:25 GMT

Previously was on CDH 4.7.1 with SCM 4.8.6 and did not experience this problem.

Currently testing CDH 5.5.1 with SCM 5.5.1. Hive Metastore is running on same host as Oracle, started via service cloudera-scm-agent on runlevels 345.

After a reboot, hive metastore gets stuck:

[main]: Failed initialising database.
Unable to open a test connection to the given database. JDBC url = jdbc:oracle:thin:@//i11-db-01:1521/odpdd, username = hiveuser. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

Once in this condition I can restart the 'hive1' service via the SCM portal, then impala and the SCM portal return to functioning properly.

As a test I disabled cloudera-scm-agent with:

# chkconfig cloudera-scm-agent off

Then did a reboot, monitored oracle connections available from a separate workstation using 'SQuirreL SQL Client' and once succeeded performed:

# service cloudera-scm-agent start

Came up fine.

We guess that even though Oracle starts on runlevel 2, it is not ready for connections when hive metastore begins, and hive metastore not gracefully handling this scenario.

Any suggestions?

What about this 'lazyinit' parameter?

Thank you.

Apr 11, 2:12:30.623 PM 	INFO 	org.apache.hadoop.hive.metastore.HiveMetaStore 	

[main]: Starting hive metastore on port 9083

Apr 11, 2:12:34.235 PM 	INFO 	org.apache.hadoop.hive.metastore.HiveMetaStore 	

[main]: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

Apr 11, 2:12:35.866 PM 	INFO 	org.apache.hadoop.hive.metastore.ObjectStore 	

[main]: ObjectStore, initialize called

Apr 11, 2:12:37.265 PM 	INFO 	DataNucleus.Persistence 	

[main]: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored

Apr 11, 2:12:37.265 PM 	INFO 	DataNucleus.Persistence 	

[main]: Property datanucleus.cache.level2 unknown - will be ignored

Apr 11, 2:12:40.032 PM 	ERROR 	DataNucleus.Datastore.Schema 	

[main]: Failed initialising database.
Unable to open a test connection to the given database. JDBC url = jdbc:oracle:thin:@//i11-db-01:1521/odpdd, username = hiveuser. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:419)
	at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:536)
	at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:228)
	at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
	at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
	at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
	at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:120)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:501)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:298)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Unknown Source)
	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
	at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
	at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:406)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:435)
	at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:330)
	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:286)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:56)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:65)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:596)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:574)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:623)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:464)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5775)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5770)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6022)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5947)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
	at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:375)
	at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:422)
	at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:678)
	at oracle.net.ns.NSProtocol.connect(NSProtocol.java:238)
	at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1054)
	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:308)
	... 56 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
	at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
	at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.net.SocksSocketImpl.connect(Unknown Source)
	at java.net.Socket.connect(Unknown Source)
	at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:209)
	at oracle.net.nt.ConnOption.connect(ConnOption.java:123)
	at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:353)
	... 61 more
------

Re: Hive Metastore fails to start properly after reboot

ScottChris — Tue, 14 Jun 2016 14:38:22 GMT

CDH 5.7.0 is not having this issue.

question Re: Hive Metastore fails to start properly after reboot in Archives of Support Questions (Read Only)

Hive Metastore fails to start properly after reboot

Re: Hive Metastore fails to start properly after reboot