Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Upgraded MySQL failed start mysql services

Labels:
TanChoonKiat
Explorer

Created on ‎01-13-2020 08:04 AM - last edited on ‎01-13-2020 12:27 PM by Moderator

Could someone help to advice on this ?

After upgrade mysql to latest version, cloudera-scm-server failed to start.

 

 

 

 

/var/log/cloudera-scm-server/cloudera-scm-server.log
2020-01-13 23:58:49,993 WARN main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: SQL Error: 0, SQLState: null
2020-01-13 23:58:49,994 ERROR main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: Connections could not be acquired from the underlying database!
2020-01-13 23:58:50,008 INFO main:org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@6200f9cb: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy
2020-01-13 23:58:50,008 ERROR main:com.cloudera.server.cmf.Main: Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
        at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:393)
        at com.cloudera.server.cmf.Main.<init>(Main.java:243)
        at com.cloudera.server.cmf.Main.main(Main.java:216)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
        ... 17 more
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)
        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)
        at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1397)
        at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:62)
        at com.cloudera.enterprise.AbstractWrappedEntityManager.beginForRollbackAndReadonly(AbstractWrappedEntityManager.java:89)
        at com.cloudera.enterprise.dbutil.DbUtil.isInnoDbEnabled(DbUtil.java:549)
        at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.checkMysqlTableEngineType(EntityManagerFactoryBean.java:139)
        at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:122)
        at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:65)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
        ... 22 more
Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
        at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:221)
        at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)
        at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
        at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
        at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1426)
        at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:59)
        ... 28 more
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
        at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)
        at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
        at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:84)
        at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
        at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214)
        ... 33 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
        at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
        at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
        at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
        ... 37 more

 

 

 

 

1,504 Views
0 Kudos
5 REPLIES 5

lwang
Super Collaborator

Created ‎01-13-2020 10:10 AM

Hi @TanChoonKiat ,

 

Could you please share the versions of your Cloudera Manager and previous/current MySQL versions? Here is the official doc on what MySQL versions that CM6.x supports:

https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_database_requirements.h...

 

Also, please verify the db host and credential configured for CM in /etc/cloudera-scm-server/db.properties on the CM host. Make sure to check if this has been configured correctly.

 

Thanks,

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

1,484 Views
0 Kudos

TanChoonKiat
Explorer

Created ‎01-13-2020 10:24 AM

Could you refer the error messages as below. I had rollback to previous version of the database and checked the db.properties. currently cloudera-scm-server up and running but still getting some error.

 

2020-01-14 02:22:42,620 WARN com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.command.flow.CmdStep: Unexpected exception during command work
java.lang.NullPointerException
        at com.cloudera.cmf.service.ScheduledSnapshotsCmdWork.doWork(ScheduledSnapshotsCmdWork.java:128)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
        at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:845)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:840)
        at com.cloudera.server.cmf.components.OperationsManagerImpl.executeServiceCmd(OperationsManagerImpl.java:1790)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.dispatchCommand(CommandDispatcherJob.java:234)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.executeCommand(CommandDispatcherJob.java:191)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.access$500(CommandDispatcherJob.java:61)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:168)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:165)
        at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
        at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210)
        at com.cloudera.server.common.RetryUtils.executeWithRetryConstantSleep(RetryUtils.java:73)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.execute(CommandDispatcherJob.java:162)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:206)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:548)
2020-01-14 02:22:42,620 WARN com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.command.flow.CmdStep: Unexpected exception during command work
java.lang.NullPointerException
        at com.cloudera.cmf.service.ScheduledSnapshotsCmdWork.doWork(ScheduledSnapshotsCmdWork.java:128)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
        at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:845)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:840)
        at com.cloudera.server.cmf.components.OperationsManagerImpl.executeServiceCmd(OperationsManagerImpl.java:1790)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.dispatchCommand(CommandDispatcherJob.java:234)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.executeCommand(CommandDispatcherJob.java:191)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.access$500(CommandDispatcherJob.java:61)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:168)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:165)
        at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
        at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210)
        at com.cloudera.server.common.RetryUtils.executeWithRetryConstantSleep(RetryUtils.java:73)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.execute(CommandDispatcherJob.java:162)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:206)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:548)
2020-01-14 02:22:42,621 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.scheduler.CommandDispatcherJob: Failed to invoke command (HdfsScheduledSnapshotsCommand) for schedule (8)
com.cloudera.cmf.command.CmdExecException: java.lang.NullPointerException
        at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:119)
        at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:845)
        at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:840)
        at com.cloudera.server.cmf.components.OperationsManagerImpl.executeServiceCmd(OperationsManagerImpl.java:1790)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.dispatchCommand(CommandDispatcherJob.java:234)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.executeCommand(CommandDispatcherJob.java:191)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.access$500(CommandDispatcherJob.java:61)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:168)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:165)
        at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
        at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210)
        at com.cloudera.server.common.RetryUtils.executeWithRetryConstantSleep(RetryUtils.java:73)
        at com.cloudera.cmf.scheduler.CommandDispatcherJob.execute(CommandDispatcherJob.java:162)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:206)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:548)
Caused by: java.lang.NullPointerException
        at com.cloudera.cmf.service.ScheduledSnapshotsCmdWork.doWork(ScheduledSnapshotsCmdWork.java:128)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
        at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
        at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
        ... 16 more
2020-01-14 02:22:42,624 INFO com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command GlobalPoolsRefresh BasicCmdArgs{scheduleId=1, scheduledTime=2016-04-03T00:00:00.000Z}.
1,480 Views
0 Kudos

lwang
Super Collaborator

Created ‎01-13-2020 11:32 AM

Hi @TanChoonKiat ,

 

How did you rollback MySQL? Did you rollback to previous "back up" database? Did you shutdown the cluster when you did the database upgrade? From the error message you posted in latest post, I feel there are some out of sync issues between the CM database and the cluster activities.

 

Thanks,

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

1,454 Views
0 Kudos

TanChoonKiat
Explorer

Created ‎01-13-2020 11:41 AM

How did you rollback MySQL?

downgrade from version 5.7 to 5.5

Did you rollback to previous "back up" database?

No backup had been done.

Did you shutdown the cluster when you did the database upgrade?

Yes, before upgrade we shut down all the cluster and services.

From the error message you posted in latest post, I feel there are some out of sync issues between the CM database and the cluster activities.

What the action could be taken ?

1,450 Views
0 Kudos

Gyan57
Cloudera Employee

Created ‎01-13-2020 11:45 AM

Hi,

 

The error message below 

 

2020-01-14 02:22:42,621 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.scheduler.CommandDispatcherJob: Failed to invoke command (HdfsScheduledSnapshotsCommand) for schedule (8) com.cloudera.cmf.command.CmdExecException: java.lang.NullPointerException

 

Suggest that there is some value in the scheduled  HDFS snapshot replication which is all and causing issue. You might even have issue when you try to browse the replication schedule pages. 

 

You might have to remove the snapshot for HDFS if you have any and re-enable the snapshot. 

 

Please browser to "Backups->Snapshot policies" and check if you have any. Try to remove them and re-add them. 

1,449 Views
0 Kudos
Take a Tour of the Community
Don't have an account?
Register
Your experience may be limited. Sign in to explore more.
Announcements
What's New @ Cloudera
Cloudera Operational Database (COD) UI supports creating a smaller cluster using a predefined Data Lake template
What's New @ Cloudera
Cloudera Operational Database (COD) supports scaling up the clusters vertically
Product Announcements
CDP Public Cloud: April 2023 Release Summary
What's New @ Cloudera
Cloudera Machine Learning launches "Add Data" feature to simplify data ingestion
What's New @ Cloudera
Simplify Data Access with Custom Connection Support in CML
View More Announcements