Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Upgraded MySQL failed start mysql services

Upgraded MySQL failed start mysql services

Could someone help to advice on this ?

After upgrade mysql to latest version, cloudera-scm-server failed to start.

 

 

 

 

/var/log/cloudera-scm-server/cloudera-scm-server.log
2020-01-13 23:58:49,993 WARN main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: SQL Error: 0, SQLState: null
2020-01-13 23:58:49,994 ERROR main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: Connections could not be acquired from the underlying database!
2020-01-13 23:58:50,008 INFO main:org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@6200f9cb: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy
2020-01-13 23:58:50,008 ERROR main:com.cloudera.server.cmf.Main: Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
        at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:393)
        at com.cloudera.server.cmf.Main.<init>(Main.java:243)
        at com.cloudera.server.cmf.Main.main(Main.java:216)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
        ... 17 more
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)
        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)
        at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1397)
        at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:62)
        at com.cloudera.enterprise.AbstractWrappedEntityManager.beginForRollbackAndReadonly(AbstractWrappedEntityManager.java:89)
        at com.cloudera.enterprise.dbutil.DbUtil.isInnoDbEnabled(DbUtil.java:549)
        at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.checkMysqlTableEngineType(EntityManagerFactoryBean.java:139)
        at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:122)
        at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:65)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
        ... 22 more
Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
        at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:221)
        at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)
        at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
        at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
        at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1426)
        at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:59)
        ... 28 more
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
        at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)
        at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
        at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:84)
        at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
        at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214)
        ... 33 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
        at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
        at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
        at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
        ... 37 more

 

 

 

 

5 REPLIES 5
Highlighted

Re: Upgraded MySQL failed start mysql services

Super Collaborator

Hi @TanChoonKiat ,

 

Could you please share the versions of your Cloudera Manager and previous/current MySQL versions? Here is the official doc on what MySQL versions that CM6.x supports:

https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_database_requirements.h...

 

Also, please verify the db host and credential configured for CM in /etc/cloudera-scm-server/db.properties on the CM host. Make sure to check if this has been configured correctly.

 

Thanks,

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Highlighted

Re: Upgraded MySQL failed start mysql services

Could you refer the error messages as below. I had rollback to previous version of the database and checked the db.properties. currently cloudera-scm-server up and running but still getting some error.

 

2020-01-14 02:22:42,620 WARN com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.command.flow.CmdStep: Unexpected exception during command work
java.lang.NullPointerException
        at com.cloudera.cmf.service.ScheduledSnapshotsCmdWork.doWork(ScheduledSnapshotsCmdWork.java:128)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
        at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:845)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:840)
        at com.cloudera.server.cmf.components.OperationsManagerImpl.executeServiceCmd(OperationsManagerImpl.java:1790)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.dispatchCommand(CommandDispatcherJob.java:234)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.executeCommand(CommandDispatcherJob.java:191)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.access$500(CommandDispatcherJob.java:61)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:168)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:165)
        at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
        at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210)
        at com.cloudera.server.common.RetryUtils.executeWithRetryConstantSleep(RetryUtils.java:73)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.execute(CommandDispatcherJob.java:162)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:206)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:548)
2020-01-14 02:22:42,620 WARN com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.command.flow.CmdStep: Unexpected exception during command work
java.lang.NullPointerException
        at com.cloudera.cmf.service.ScheduledSnapshotsCmdWork.doWork(ScheduledSnapshotsCmdWork.java:128)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
        at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:845)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:840)
        at com.cloudera.server.cmf.components.OperationsManagerImpl.executeServiceCmd(OperationsManagerImpl.java:1790)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.dispatchCommand(CommandDispatcherJob.java:234)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.executeCommand(CommandDispatcherJob.java:191)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.access$500(CommandDispatcherJob.java:61)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:168)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:165)
        at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
        at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210)
        at com.cloudera.server.common.RetryUtils.executeWithRetryConstantSleep(RetryUtils.java:73)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.execute(CommandDispatcherJob.java:162)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:206)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:548)
2020-01-14 02:22:42,621 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.scheduler.CommandDispatcherJob: Failed to invoke command (HdfsScheduledSnapshotsCommand) for schedule (8)
com.cloudera.cmf.command.CmdExecException: java.lang.NullPointerException
        at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:119)
        at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:845)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:840)
        at com.cloudera.server.cmf.components.OperationsManagerImpl.executeServiceCmd(OperationsManagerImpl.java:1790)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.dispatchCommand(CommandDispatcherJob.java:234)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.executeCommand(CommandDispatcherJob.java:191)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.access$500(CommandDispatcherJob.java:61)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:168)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:165)
        at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
        at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210)
        at com.cloudera.server.common.RetryUtils.executeWithRetryConstantSleep(RetryUtils.java:73)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.execute(CommandDispatcherJob.java:162)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:206)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:548)
Caused by: java.lang.NullPointerException
        at com.cloudera.cmf.service.ScheduledSnapshotsCmdWork.doWork(ScheduledSnapshotsCmdWork.java:128)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
        ... 16 more
2020-01-14 02:22:42,624 INFO com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command GlobalPoolsRefresh BasicCmdArgs{scheduleId=1, scheduledTime=2016-04-03T00:00:00.000Z}.
Highlighted

Re: Upgraded MySQL failed start mysql services

Super Collaborator

Hi @TanChoonKiat ,

 

How did you rollback MySQL? Did you rollback to previous "back up" database? Did you shutdown the cluster when you did the database upgrade? From the error message you posted in latest post, I feel there are some out of sync issues between the CM database and the cluster activities.

 

Thanks,

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Highlighted

Re: Upgraded MySQL failed start mysql services

How did you rollback MySQL?

downgrade from version 5.7 to 5.5

Did you rollback to previous "back up" database?

No backup had been done.

Did you shutdown the cluster when you did the database upgrade?

Yes, before upgrade we shut down all the cluster and services.

From the error message you posted in latest post, I feel there are some out of sync issues between the CM database and the cluster activities.

What the action could be taken ?

Highlighted

Re: Upgraded MySQL failed start mysql services

Cloudera Employee

Hi,

 

The error message below 

 

2020-01-14 02:22:42,621 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.scheduler.CommandDispatcherJob: Failed to invoke command (HdfsScheduledSnapshotsCommand) for schedule (8) com.cloudera.cmf.command.CmdExecException: java.lang.NullPointerException

 

Suggest that there is some value in the scheduled  HDFS snapshot replication which is all and causing issue. You might even have issue when you try to browse the replication schedule pages. 

 

You might have to remove the snapshot for HDFS if you have any and re-enable the snapshot. 

 

Please browser to "Backups->Snapshot policies" and check if you have any. Try to remove them and re-add them. 

Don't have an account?
Coming from Hortonworks? Activate your account here