Can't start cloudera-scm-server after upgrading to 5.14.2

[#1]

Hi,

I upgraded my CDH from 5.14.1 to 5.14.2 and now I can't start cloudera-scm-server. It gives me this error:

2018-04-15 12:23:44,501 INFO main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: MYSQL database engine and table mapping: {InnoDB=[AUDITS, CLIENT_CONFIGS, EXTERNAL_ACCOUNTS_AUD, PROCESSES_DETAIL, RELEASES, RELEASES_AUD, SERVICES, USER_SETTINGS, EXTERNAL_ACCOUNTS, HOSTS_AUD, HOST_TEMPLATE_TO_ROLE_CONF_GRP, DIAGNOSTICS_EVENTS, CM_PEERS, CLUSTER_ACTIVATED_RELEASES, SCHEMA_VERSION, CREDENTIALS, ROLES_AUD, PARCEL_COMPONENTS, ROLE_CONFIG_GROUPS, ROLE_STALENESS_STATUS, CLUSTERS_AUD, SERVICES_AUD, COMMAND_SCHEDULES, CONFIGS_AUD, SNAPSHOT_POLICIES, REVISIONS, CLUSTER_MANAGED_RELEASES, CONFIG_CONTAINERS, GLOBAL_SETTINGS, METRICS, HOST_TEMPLATES, PARCELS, PROCESSES, COMMANDS_DETAIL, CLUSTER_UNDISTRIBUTED_RELEASES, CONFIGS, USER_ROLES, ROLE_CONFIG_GROUPS_AUD, HOSTS, CLIENT_CONFIGS_TO_HOSTS, COMMANDS, CLUSTERS, ROLES, CLUSTER_ACTIVATED_RELEASES_AUD, USERS, CM_VERSION, PROCESS_ACTIVE_RELEASES]}
2018-04-15 12:23:44,591 ERROR main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean:
********************************************************************************
* Cloudera Manager cannot be safely upgraded while there are pending           *
* active commands. Please downgrade to the prior version 5.14.1                *
* and either let the commands run to completion or abort them.                 *
* There are 6 active commands of type Start                                    *
*                                                                              *
* If you must proceed without downgrade, active commands can be hard           *
* stopped if you start cloudera-scm-server with 'force_start' argument,        *
* e.g. 'service cloudera-scm-server force_start'                               *
********************************************************************************

2018-04-15 12:23:44,593 INFO main:org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7dc0f706: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy
2018-04-15 12:23:44,594 ERROR main:com.cloudera.server.cmf.Main: Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Cloudera Manager cannot be upgraded with active commands
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
        at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:393)
        at com.cloudera.server.cmf.Main.<init>(Main.java:243)
        at com.cloudera.server.cmf.Main.main(Main.java:216)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Cloudera Manager cannot be upgraded with active commands
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
        ... 17 more
Caused by: java.lang.RuntimeException: Cloudera Manager cannot be upgraded with active commands
        at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.failIfActiveCommandsExist(EntityManagerFactoryBean.java:229)
        at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.ensureNoActiveCommandsDuringUpgrade(EntityManagerFactoryBean.java:190)
        at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:124)
        at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:65)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
        ... 22 more

I have Ubuntu 16.4 and the external database MySQL.

Many thanks.

[#2]

As an update I checked `/etc/cloudera-scm-server/db.properties` and it is correct and I can connect to the scm database without a problem with username/password.

[#3 · Accepted solution]

I followed the instruction of downgrading back to 5.14.1 on Ubuntu from here: https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_ag_downgrade_to_cm4.html

The server started without a problem, I upgraded CM to 5.14.2 on all the masters again and on the Web console, I upgraded agents to 5.14.2 on all the machines.

I am guessing an interruption caused by the unexpected shutdown or a leftover of hard-coded apt pointing to 5.14.1 may have caused this. But now everything has been resolved.

Thank you.