Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

cloudera 5.7.0 + After OS ( ubuntu 14.04) upgrade fail to start cloudera-scm-server

cloudera 5.7.0 + After OS ( ubuntu 14.04) upgrade fail to start cloudera-scm-server

Hi

I upgarde my ubuntu 14.04 system using

   apt-get update

   apt-get upgrade

 

after that my cloudera-scm-server failed to start . i checked cloudera-scm-server status

          $  sudo service cloudera-scm-server status

                  Checking for service cloudera-scm-server: * cloudera-scm-server is dead and pid file exists

 

this scenerio  comes when i did  "restart my cloudera services " and "upgradation of my OS and restart " in parallel.

 

logs  of /var/log/cloudera-scm-server/cloudera-scm-server.log : 

 

2016-06-06 15:47:44,606 ERROR main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: ERROR: column "success" is of type boolean but expression is of type integer
Hint: You will need to rewrite or cast the expression.
Position: 43
2016-06-06 15:47:44,617 WARN main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Failed to hard stop active commands
javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not execute statement
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1397)
at org.hibernate.ejb.AbstractQueryImpl.executeUpdate(AbstractQueryImpl.java:108)
at com.cloudera.cmf.persist.DbCommandDao2.hardStopActiveCommands(DbCommandDao2.java:644)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.hardStopActiveCommands(EntityManagerFactoryBean.java:233)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.ensureNoActiveCommandsDuringUpgrade(EntityManagerFactoryBean.java:182)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:123)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:64)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:387)
at com.cloudera.server.cmf.Main.<init>(Main.java:242)
at com.cloudera.server.cmf.Main.main(Main.java:216)

Caused by: org.hibernate.exception.SQLGrammarException: could not execute statement
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:123)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:136)
at org.hibernate.engine.query.spi.NativeSQLQueryPlan.performExecuteUpdate(NativeSQLQueryPlan.java:204)
at org.hibernate.internal.SessionImpl.executeNativeUpdate(SessionImpl.java:1313)
at org.hibernate.internal.SQLQueryImpl.executeUpdate(SQLQueryImpl.java:401)
at org.hibernate.ejb.QueryImpl.internalExecuteUpdate(QueryImpl.java:194)
at org.hibernate.ejb.AbstractQueryImpl.executeUpdate(AbstractQueryImpl.java:99)
... 28 more
Caused by: org.postgresql.util.PSQLException: ERROR: column "success" is of type boolean but expression is of type integer
Hint: You will need to rewrite or cast the expression.
Position: 43
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:133)
... 33 more
2016-06-06 15:47:44,620 INFO main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Hard stopped 0 commands
2016-06-06 15:47:44,633 ERROR main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean:
********************************************************************************
* Cloudera Manager cannot be safely upgraded while there are pending *
* active commands. Please downgrade to the prior version 5.7 *
* and either let the commands run to completion or abort them. *
* There are 5 active commands of type Start *
* There are 1 active commands of type YarnOrderedStart *
* There are 2 active commands of type Restart *
* *
* If you must proceed without downgrade, active commands can be hard *
* stopped if you start cloudera-scm-server with 'force_start' argument, *
* e.g. 'service cloudera-scm-server force_start' *
********************************************************************************

 

 

2016-06-06 15:47:44,635 INFO main:org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@695d7d60: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy
2016-06-06 15:47:44,635 ERROR main:com.cloudera.server.cmf.Main: Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Cloudera Manager cannot be upgraded with active commands
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:387)
at com.cloudera.server.cmf.Main.<init>(Main.java:242)
at com.cloudera.server.cmf.Main.main(Main.java:216)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Cloudera Manager cannot be upgraded with active commands
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 17 more
Caused by: java.lang.RuntimeException: Cloudera Manager cannot be upgraded with active commands
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.failIfActiveCommandsExist(EntityManagerFactoryBean.java:225)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.ensureNoActiveCommandsDuringUpgrade(EntityManagerFactoryBean.java:186)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:123)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:64)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
... 22 more

 

 

 

when i checked the cloudera services version its show 5.7.1 but he should be 5.7.0

 

$ dpkg-query -l 'cloudera-manager-*'


Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===========================================-==========================-==========================-===========================================================================================
ii cloudera-manager-agent 5.7.1-1.cm571.p0.8~trusty- amd64 The Cloudera Manager Agent
ii cloudera-manager-daemons 5.7.1-1.cm571.p0.8~trusty- all Provides daemons for monitoring Hadoop and related tools.
ii cloudera-manager-repository 5.0 all Cloudera Manager
ii cloudera-manager-server 5.7.1-1.cm571.p0.8~trusty- all The Cloudera Manager Server
un cloudera-manager-server-db <none> <none> (no description available)
ii cloudera-manager-server-db-2 5.7.1-1.cm571.p0.8~trusty- all This package configures an "embedded" PostgreSQL server,running as user cloudera-scm on por

 

entry inside /etc/apt/sources.list.d/cloudera-manager.list is

  deb [arch=amd64] https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/ trusty-cm5.7.0 contrib

 

 

please help me to solve the issue .

 

Regards

Prateek