Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie startup failure: ERROR XSDB6

Oozie startup failure: ERROR XSDB6

New Contributor

Hi,

 

I am running into the following issue when starting the Oozie service through ClouderaManager:

 

SERVER[hadoop-datanode-00.qa.wh.reachlocal.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] E0103: Could not load service classes, Cannot create PoolableConnectionFactory (Failed to start database '/var/lib/oozie/data' with class loader org.apache.catalina.loader.StandardClassLoader@301875e1, see the next exception for details.)
org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Cannot create PoolableConnectionFactory (Failed to start database '/var/lib/oozie/data' with class loader org.apache.catalina.loader.StandardClassLoader@301875e1, see the next exception for details.)
	at org.apache.oozie.service.Services.loadServices(Services.java:291)
	at org.apache.oozie.service.Services.init(Services.java:208)
	at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:45)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
	at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
	at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
	at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
	at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
	at org.apache.catalina.core.StandardService.start(StandardService.java:525)
	at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: <openjpa-2.2.2-r422266:1468616 fatal general error> org.apache.openjpa.persistence.PersistenceException: Cannot create PoolableConnectionFactory (Failed to start database '/var/lib/oozie/data' with class loader org.apache.catalina.loader.StandardClassLoader@301875e1, see the next exception for details.)
	at org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:102)
	at org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDBDictionaryInstance(JDBCConfigurationImpl.java:603)
	at org.apache.openjpa.jdbc.meta.MappingRepository.endConfiguration(MappingRepository.java:1518)
	at org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:531)
	at org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:456)
	at org.apache.openjpa.lib.conf.PluginValue.instantiate(PluginValue.java:120)
	at org.apache.openjpa.conf.MetaDataRepositoryValue.instantiate(MetaDataRepositoryValue.java:68)
	at org.apache.openjpa.lib.conf.ObjectValue.instantiate(ObjectValue.java:83)
	at org.apache.openjpa.conf.OpenJPAConfigurationImpl.newMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:967)
	at org.apache.openjpa.conf.OpenJPAConfigurationImpl.getMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:958)
	at org.apache.openjpa.kernel.AbstractBrokerFactory.makeReadOnly(AbstractBrokerFactory.java:644)
	at org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:203)
	at org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:156)
	at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:227)
	at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:154)
	at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:60)
	at org.apache.oozie.service.JPAService.getEntityManager(JPAService.java:467)
	at org.apache.oozie.service.JPAService.init(JPAService.java:168)
	at org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
	at org.apache.oozie.service.Services.setService(Services.java:354)
	at org.apache.oozie.service.Services.loadServices(Services.java:287)
	... 26 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Failed to start database '/var/lib/oozie/data' with class loader org.apache.catalina.loader.StandardClassLoader@301875e1, see the next exception for details.)
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
	at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
	at org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)
	at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87)
	at org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:91)
	... 46 more
Caused by: java.sql.SQLException: Failed to start database '/var/lib/oozie/data' with class loader org.apache.catalina.loader.StandardClassLoader@301875e1, see the next exception for details.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source)
	at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
	at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
	at org.apache.derby.jdbc.Driver20.connect(Unknown Source)
	at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
	at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
	at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
	at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
	... 51 more
Caused by: java.sql.SQLException: Failed to start database '/var/lib/oozie/data' with class loader org.apache.catalina.loader.StandardClassLoader@301875e1, see the next exception for details.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
	... 65 more
Caused by: java.sql.SQLException: Another instance of Derby may have already booted the database /var/lib/oozie/data.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
	... 62 more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /var/lib/oozie/data.
	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.getJBMSLockOnDB(Unknown Source)
	at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
	at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
	at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
	at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
	at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
	at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
	at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
	at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
	at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source)
	at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source)
	at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source)
	... 62 more

 

I have previously been able to stop/start the Oozie service successfully. The Oozie start service command only started failing recently. Not sure if this is related or not, but just before this error started surfacing, Oozie jobs were failing with a "GC limit exceeded" exception.

 

We're running a tiny (4 node) cluster on CDH5.

 

Any help would be much appreciated.

 

Thanks.

1 REPLY 1

Re: Oozie startup failure: ERROR XSDB6

Master Guru
Its likely that another instance of Oozie may be running on the machine already thats causing this. You can first check for such an instance via a "ps aux | grep oozie" command perhaps, and kill it if one is indeed running.

If there's no other instance, then its likely that the OOME caused crash/kill has left the DB lock under the derby directory, causing the further startups to fail as it now thinks the DB is in use (Derby DB supports only 1 max. current user, which is why we strongly discourage its use in your clusters - consider moving off of Derby when you get a chance).

To resolve that, run: rm /var/lib/oozie/data/*.lck

After this, start up the Oozie server again, and it should come up now.

Once again: Consider changing the default DB of Oozie to MySQL/etc. instead of Derby. You'll avoid a lot of other pains that will come later if you continue to use it.