Oozie Service (HDP 2.4.0) Failed To Start on Windows Server 2012

[#1]

Hi,

I set up a HDP cluster on VMs with OS Windows Server 2012. For Oozie set up, I am using default derby server.

However, I am not able to start the Oozieservices on Oozie host.

The service keeps restarting. And the error in oozie-error.log is attached below.

Thanks!

Jinling

2016-04-14 17:37:29,445 FATAL Services:514 - SERVER[HADOOPHDP] Runtime Exception during Services Load. Check your list of 'oozie.services' or 'oozie.services.ext'
2016-04-14 17:37:29,452 FATAL Services:514 - SERVER[HADOOPHDP] E0103: Could not load service classes, Cannot create PoolableConnectionFactory (Schema 'SA' does not exist)
org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Cannot create PoolableConnectionFactory (Schema 'SA' does not exist)
	at org.apache.oozie.service.Services.loadServices(Services.java:309)
	at org.apache.oozie.service.Services.init(Services.java:213)
	at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
	at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:676)
	at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:602)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:503)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)
	at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)
	at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
	at org.apache.catalina.core.StandardService.start(StandardService.java:525)
	at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: <openjpa-2.2.2-r422266:1468616 fatal general error> org.apache.openjpa.persistence.PersistenceException: Cannot create PoolableConnectionFactory (Schema 'SA' does not exist)
	at org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:102)
	at org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDBDictionaryInstance(JDBCConfigurationImpl.java:603)
	at org.apache.openjpa.jdbc.meta.MappingRepository.endConfiguration(MappingRepository.java:1518)
	at org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:531)
	at org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:456)
	at org.apache.openjpa.lib.conf.PluginValue.instantiate(PluginValue.java:120)
	at org.apache.openjpa.conf.MetaDataRepositoryValue.instantiate(MetaDataRepositoryValue.java:68)
	at org.apache.openjpa.lib.conf.ObjectValue.instantiate(ObjectValue.java:83)
	at org.apache.openjpa.conf.OpenJPAConfigurationImpl.newMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:967)
	at org.apache.openjpa.conf.OpenJPAConfigurationImpl.getMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:958)
	at org.apache.openjpa.kernel.AbstractBrokerFactory.makeReadOnly(AbstractBrokerFactory.java:644)
	at org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:203)
	at org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:156)
	at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:227)
	at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:154)
	at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:60)
	at org.apache.oozie.service.JPAService.getEntityManager(JPAService.java:500)
	at org.apache.oozie.service.JPAService.init(JPAService.java:201)
	at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
	at org.apache.oozie.service.Services.setService(Services.java:372)
	at org.apache.oozie.service.Services.loadServices(Services.java:305)
	... 26 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Schema 'SA' does not exist)
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
	at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
	at org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)
	at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87)
	at org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:91)
	... 46 more
Caused by: java.sql.SQLSyntaxErrorException: Schema 'SA' does not exist
	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
	at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
	at org.apache.derby.impl.jdbc.EmbedStatement.executeQuery(Unknown Source)
	at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
	at org.apache.commons.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:658)
	at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1558)
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
	... 51 more
Caused by: java.sql.SQLException: Schema 'SA' does not exist
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
	at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
	... 63 more
Caused by: ERROR 42Y07: Schema 'SA' does not exist
	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
	at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(Unknown Source)
	at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(Unknown Source)
	at org.apache.derby.impl.sql.compile.QueryTreeNode.getSchemaDescriptor(Unknown Source)
	at org.apache.derby.impl.sql.compile.FromBaseTable.bindTableDescriptor(Unknown Source)
	at org.apache.derby.impl.sql.compile.FromBaseTable.bindNonVTITables(Unknown Source)
	at org.apache.derby.impl.sql.compile.FromList.bindTables(Unknown Source)
	at org.apache.derby.impl.sql.compile.SelectNode.bindNonVTITables(Unknown Source)
	at org.apache.derby.impl.sql.compile.DMLStatementNode.bindTables(Unknown Source)
	at org.apache.derby.impl.sql.compile.DMLStatementNode.bind(Unknown Source)
	at org.apache.derby.impl.sql.compile.CursorNode.bindStatement(Unknown Source)
	at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
	at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
	at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)
	... 57 more

[#2]

This error means that something is wrong with your Derby database. Can you check is it created? The path do Derby files is given by "Oozie Data Dir" (oozie_data_dir) in Ambari->Oozie->Oozie server and OOZIE_DATA in oozie-env.sh. Check permissions to that path and retry. You can also try to create the DB manually if ooziedb.bat is available, but better go through Ambari.

[#3]

Thanks very much for your reply!

I was not using Ambari, but following the steps in Installing HDP for Windows.

I checked my OOZIE_DATA path (C:\hadoop\data\oozie) and the db is created there (oozie-db).

Do you have any suggestion what should I do in this case? Does that mean the db is created but Oozie can't find it?

Or the db is created but the schema is not?

Thanks!

Jinling

[#4]

@Jinling Li - I'm not much familiar with oozie installation on Windows but did you check if oozie user has correct permission to modify oozie_db? can you please double check that? If its new installation then you can just remove oozie_db and recreate blank one with appropriate permissions and oozie-server daemon will take care of other things.

[#5]

Thanks for the suggestions!

I checked the permission under the directory and also tried recreate db again. But the same problem occurred..I now changed to use MS SQL server, and it's working perfectly now.

[#6 · Accepted solution]

replace derby database with sql server, there are multiple questions with same issue on HCC and all stem from using Derby with Oozie on Windows.

[#7]

Thanks! I saw several posts reporting issues with Derby+Oozie and Derby+Hive on Windows. I re-configured the cluster and used MS SQL server and they work perfectly now.