Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive Metastore fails to start properly after reboot

avatar
Contributor

Previously was on CDH 4.7.1 with SCM 4.8.6 and did not experience this problem.

 

Currently testing CDH 5.5.1 with SCM 5.5.1.  Hive Metastore is running on same host as Oracle, started via service cloudera-scm-agent on runlevels 345.

 

After a reboot, hive metastore gets stuck:

 

[main]: Failed initialising database.
Unable to open a test connection to the given database. JDBC url = jdbc:oracle:thin:@//i11-db-01:1521/odpdd, username = hiveuser. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

 

Once in this condition I can restart the 'hive1' service via the SCM portal, then impala and the SCM portal return to functioning properly.

 

As a test I disabled cloudera-scm-agent with:

 

# chkconfig cloudera-scm-agent off

 

Then did a reboot, monitored oracle connections available from a separate workstation using 'SQuirreL SQL Client' and once succeeded performed:

 

# service cloudera-scm-agent start

 

Came up fine.

 

We guess that even though Oracle starts on runlevel 2, it is not ready for connections when hive metastore begins, and hive metastore not gracefully handling this scenario.

 

Any suggestions?

 

What about this 'lazyinit' parameter?

 

Thank you.

 

 

 

Apr 11, 2:12:30.623 PM 	INFO 	org.apache.hadoop.hive.metastore.HiveMetaStore 	

[main]: Starting hive metastore on port 9083

Apr 11, 2:12:34.235 PM 	INFO 	org.apache.hadoop.hive.metastore.HiveMetaStore 	

[main]: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

Apr 11, 2:12:35.866 PM 	INFO 	org.apache.hadoop.hive.metastore.ObjectStore 	

[main]: ObjectStore, initialize called

Apr 11, 2:12:37.265 PM 	INFO 	DataNucleus.Persistence 	

[main]: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored

Apr 11, 2:12:37.265 PM 	INFO 	DataNucleus.Persistence 	

[main]: Property datanucleus.cache.level2 unknown - will be ignored

Apr 11, 2:12:40.032 PM 	ERROR 	DataNucleus.Datastore.Schema 	

[main]: Failed initialising database.
Unable to open a test connection to the given database. JDBC url = jdbc:oracle:thin:@//i11-db-01:1521/odpdd, username = hiveuser. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:419)
	at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:536)
	at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:228)
	at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
	at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
	at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
	at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:120)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:501)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:298)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Unknown Source)
	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
	at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
	at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:406)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:435)
	at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:330)
	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:286)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:56)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:65)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:596)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:574)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:623)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:464)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5775)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5770)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6022)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5947)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
	at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:375)
	at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:422)
	at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:678)
	at oracle.net.ns.NSProtocol.connect(NSProtocol.java:238)
	at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1054)
	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:308)
	... 56 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
	at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
	at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.net.SocksSocketImpl.connect(Unknown Source)
	at java.net.Socket.connect(Unknown Source)
	at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:209)
	at oracle.net.nt.ConnOption.connect(ConnOption.java:123)
	at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:353)
	... 61 more
------

 

1 ACCEPTED SOLUTION

avatar
Contributor

CDH 5.7.0 is not having this issue.

View solution in original post

1 REPLY 1

avatar
Contributor

CDH 5.7.0 is not having this issue.