Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can't start Cloudera Manager or Hadoop Services after hard restart

Can't start Cloudera Manager or Hadoop Services after hard restart

Explorer

I am looking for advice on how to approach the problem. I have a node on which all the services are running like cloudera manager, namenode, rewsource manager, hue ,oozie and 4 other nodes as datanodes. I have seen some issues in cloudera manager, I stopped all the services in cloudera manager and hard restarted the node where all the serices are running. But once after I restarted, i see that cloiudera server is not starting showing the following log. Cloudera manager is installed using the embedded postgresql.

 

2014-06-27 09:52:19,913 WARN [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5a82c34 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:

org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)

at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)

at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)

at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)

at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)

at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)

at org.postgresql.Driver.makeConnection(Driver.java:393)

at org.postgresql.Driver.connect(Driver.java:267)

at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)

at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)

at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)

at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)

at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:579)

at java.net.Socket.connect(Socket.java:528)

at java.net.Socket.<init>(Socket.java:425)

at java.net.Socket.<init>(Socket.java:208)

at org.postgresql.core.PGStream.<init>(PGStream.java:62)

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76)

... 16 more

2014-06-27 09:52:19,927 WARN [main:internal.JdbcServicesImpl@169] HHH000342: Could not obtain connection to query metadata &colon; Connections could not be acquired from the underlying database!

2014-06-27 09:52:19,957 INFO [main:dialect.Dialect@130] HHH000400: Using dialect: org.hibernate.dialect.PostgreSQLDialect

2014-06-27 09:52:19,989 INFO [main:internal.LobCreatorBuilder@85] HHH000422: Disabling contextual LOB creation as connection was null

2014-06-27 09:52:20,656 INFO [main:internal.TransactionFactoryInitiator@73] HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory

2014-06-27 09:52:20,672 INFO [main:ast.ASTQueryTranslatorFactory@48] HHH000397: Using ASTQueryTranslatorFactory

2014-06-27 09:52:21,006 INFO [main:spi.UpdateTimestampsCache@61] HHH000250: Starting update timestamps cache at region: org.hibernate.cache.spi.UpdateTimestampsCache

2014-06-27 09:52:21,014 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.spi.UpdateTimestampsCache]; using defaults.

2014-06-27 09:52:21,023 INFO [main:internal.StandardQueryCache@86] HHH000248: Starting query cache at region: org.hibernate.cache.internal.StandardQueryCache

2014-06-27 09:52:21,025 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.internal.StandardQueryCache]; using defaults.

2014-06-27 09:52:21,128 INFO [main:util.Version@27] HV000001: Hibernate Validator 5.0.1.Final

2014-06-27 09:52:22,571 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbHost]; using defaults.

2014-06-27 09:52:22,738 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCredential]; using defaults.

2014-06-27 09:52:22,746 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbService]; using defaults.

2014-06-27 09:52:22,820 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbRoleConfigGroup]; using defaults.

2014-06-27 09:52:22,837 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCmPeer]; using defaults.

2014-06-27 09:52:22,846 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCluster]; using defaults.

2014-06-27 09:52:22,865 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbConfigContainer]; using defaults.

2014-06-27 09:52:22,904 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbHostTemplate]; using defaults.

2014-06-27 09:52:22,918 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbClientConfig]; using defaults.

2014-06-27 09:52:22,931 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbUser]; using defaults.

2014-06-27 09:52:22,941 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbGlobalSetting]; using defaults.

2014-06-27 09:52:22,947 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbRevision]; using defaults.

2014-06-27 09:52:22,993 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbUserSetting]; using defaults.

2014-06-27 09:52:23,004 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCommandSchedule]; using defaults.

2014-06-27 09:52:23,034 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbProcess]; using defaults.

2014-06-27 09:52:23,247 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbUserRole]; using defaults.

2014-06-27 09:52:23,254 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbParcel]; using defaults.

2014-06-27 09:52:23,269 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbRole]; using defaults.

2014-06-27 09:52:23,286 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbRelease]; using defaults.

2014-06-27 09:52:23,294 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbConfig]; using defaults.

2014-06-27 09:52:23,356 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbHost.roles]; using defaults.

2014-06-27 09:52:23,359 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbService.configsForDb]; using defaults.

2014-06-27 09:52:23,361 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCluster.activatedReleases]; using defaults.

2014-06-27 09:52:23,365 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbService.roleConfigGroups]; using defaults.

2014-06-27 09:52:23,368 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbConfigContainer.configsForDb]; using defaults.

2014-06-27 09:52:23,370 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbHostTemplate.roleConfigGroups]; using defaults.

2014-06-27 09:52:23,372 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbRoleConfigGroup.roles]; using defaults.

2014-06-27 09:52:23,374 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbProcess.parcels]; using defaults.

2014-06-27 09:52:23,375 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCluster.hostTemplates]; using defaults.

2014-06-27 09:52:23,932 WARN [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3a12a85e -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:

org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)

at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)

at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)

at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)

at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)

at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)

at org.postgresql.Driver.makeConnection(Driver.java:393)

at org.postgresql.Driver.connect(Driver.java:267)

at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)

at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)

at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)

at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)

at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:579)

at java.net.Socket.connect(Socket.java:528)

at java.net.Socket.<init>(Socket.java:425)

at java.net.Socket.<init>(Socket.java:208)

at org.postgresql.core.PGStream.<init>(PGStream.java:62)

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76)

... 16 more

2014-06-27 09:52:23,935 WARN [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@a03330d -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:

org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)

at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)

at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)

at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)

at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)

at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)

at org.postgresql.Driver.makeConnection(Driver.java:393)

at org.postgresql.Driver.connect(Driver.java:267)

at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)

at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)

at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)

at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)

at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:579)

at java.net.Socket.connect(Socket.java:528)

at java.net.Socket.<init>(Socket.java:425)

at java.net.Socket.<init>(Socket.java:208)

at org.postgresql.core.PGStream.<init>(PGStream.java:62)

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76)

... 16 more

2014-06-27 09:52:24,732 INFO [main:enterprise.CommonMain@240] Statistics not enabled, JMX will not be registered

2014-06-27 09:52:28,904 WARN [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@66d182d7 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:

org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)

at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)

at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)

at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)

at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)

at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)

at org.postgresql.Driver.makeConnection(Driver.java:393)

at org.postgresql.Driver.connect(Driver.java:267)

at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)

at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)

at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)

at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)

at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:579)

at java.net.Socket.connect(Socket.java:528)

at java.net.Socket.<init>(Socket.java:425)

at java.net.Socket.<init>(Socket.java:208)

at org.postgresql.core.PGStream.<init>(PGStream.java:62)

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76)

... 16 more

2014-06-27 09:52:28,905 WARN [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@60e208c1 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:

org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)

at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)

at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)

at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)

at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)

at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)

at org.postgresql.Driver.makeConnection(Driver.java:393)

at org.postgresql.Driver.connect(Driver.java:267)

at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)

at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)

at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)

at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)

at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:579)

at java.net.Socket.connect(Socket.java:528)

at java.net.Socket.<init>(Socket.java:425)

at java.net.Socket.<init>(Socket.java:208)

at org.postgresql.core.PGStream.<init>(PGStream.java:62)

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76)

... 16 more

2014-06-27 09:52:28,904 WARN [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@44e525fd -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:

org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)

at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)

at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)

at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)

at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)

at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)

at org.postgresql.Driver.makeConnection(Driver.java:393)

at org.postgresql.Driver.connect(Driver.java:267)

at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)

at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)

at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)

at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)

at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:579)

at java.net.Socket.connect(Socket.java:528)

at java.net.Socket.<init>(Socket.java:425)

at java.net.Socket.<init>(Socket.java:208)

at org.postgresql.core.PGStream.<init>(PGStream.java:62)

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76)

... 16 more

2014-06-27 09:52:28,907 WARN [main:spi.SqlExceptionHelper@145] SQL Error: 0, SQLState: null

2014-06-27 09:52:28,914 ERROR [main:spi.SqlExceptionHelper@147] Connections could not be acquired from the underlying database!

2014-06-27 09:52:28,923 INFO [main:support.DefaultListableBeanFactory@422] Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4bdc0b6b: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy

2014-06-27 09:52:28,924 ERROR [main:cmf.Main@201] Server failed.

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is org.hibernate.exception.GenericJDBCException: Could not open connection

at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)

at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)

at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)

at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)

at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)

at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)

at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:352)

at com.cloudera.server.cmf.Main.<init>(Main.java:213)

at com.cloudera.server.cmf.Main.main(Main.java:196)

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is org.hibernate.exception.GenericJDBCException: Could not open connection

at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)

at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)

at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)

at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)

... 17 more

Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection

at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)

at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)

at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)

at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:221)

at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)

at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:282)

at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2000)

at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:1986)

at com.cloudera.enterprise.dbutil.DbUtil.getSchemaVersion(DbUtil.java:183)

at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.checkVersionDoFail(EntityManagerFactoryBean.java:158)

at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:104)

at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:50)

at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)

... 22 more

Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!

at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)

at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)

at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:84)

at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)

at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214)

... 31 more

Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.

at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)

at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)

at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)

... 35 more

2014-06-27 11:05:50,156 INFO [main:mortbay.log@67] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog

2014-06-27 11:05:50,159 INFO [main:mortbay.log@67] Registered SubjectType HDFS

2014-06-27 11:05:50,163 INFO [main:mortbay.log@67] Registered SubjectType HDFS-NAMENODE

2014-06-27 11:05:50,167 INFO [main:mortbay.log@67] Registered SubjectType HDFS-DATANODE

2014-06-27 11:05:50,168 INFO [main:mortbay.log@67] Registered SubjectType HDFS-SECONDARYNAMENODE

2014-06-27 11:05:50,169 INFO [main:mortbay.log@67] Registered SubjectType HDFS-JOURNALNODE

2014-06-27 11:05:50,170 INFO [main:mortbay.log@67] Registered SubjectType HDFS-HTTPFS

2014-06-27 11:05:50,171 INFO [main:mortbay.log@67] Registered SubjectType HDFS-FAILOVERCONTROLLER

2014-06-27 11:05:50,172 INFO [main:mortbay.log@67] Registered SubjectType HDFS-NFSGATEWAY

2014-06-27 11:05:50,172 INFO [main:mortbay.log@67] Registered SubjectType HBASE

2014-06-27 11:05:50,173 INFO [main:mortbay.log@67] Registered SubjectType HBASE-MASTER

2014-06-27 11:05:50,174 INFO [main:mortbay.log@67] Registered SubjectType HBASE-REGIONSERVER

2014-06-27 11:05:50,174 INFO [main:mortbay.log@67] Registered SubjectType HBASE-HBASETHRIFTSERVER

2014-06-27 11:05:50,175 INFO [main:mortbay.log@67] Registered SubjectType HBASE-HBASERESTSERVER

2014-06-27 11:05:50,175 INFO [main:mortbay.log@67] Registered SubjectType MAPREDUCE

2014-06-27 11:05:50,176 INFO [main:mortbay.log@67] Registered SubjectType MAPREDUCE-JOBTRACKER

2014-06-27 11:05:50,182 INFO [main:mortbay.log@67] Registered SubjectType MAPREDUCE-TASKTRACKER

2014-06-27 11:05:50,183 INFO [main:mortbay.log@67] Registered SubjectType MAPREDUCE-FAILOVERCONTROLLER

2014-06-27 11:05:50,184 INFO [main:mortbay.log@67] Registered SubjectType MGMT

2014-06-27 11:05:50,185 INFO [main:mortbay.log@67] Registered SubjectType MGMT-ACTIVITYMONITOR

2014-06-27 11:05:50,186 INFO [main:mortbay.log@67] Registered SubjectType MGMT-SERVICEMONITOR

2014-06-27 11:05:50,187 INFO [main:mortbay.log@67] Registered SubjectType MGMT-HOSTMONITOR

2014-06-27 11:05:50,188 INFO [main:mortbay.log@67] Registered SubjectType MGMT-EVENTSERVER

2014-06-27 11:05:50,188 INFO [main:mortbay.log@67] Registered SubjectType MGMT-ALERTPUBLISHER

2014-06-27 11:05:50,189 INFO [main:mortbay.log@67] Registered SubjectType MGMT-REPORTSMANAGER

2014-06-27 11:05:50,190 INFO [main:mortbay.log@67] Registered SubjectType MGMT-NAVIGATOR

2014-06-27 11:05:50,191 INFO [main:mortbay.log@67] Registered SubjectType MGMT-NAVIGATORMETASERVER

2014-06-27 11:05:50,191 INFO [main:mortbay.log@67] Registered SubjectType HOST

2014-06-27 11:05:50,191 INFO [main:mortbay.log@67] Registered SubjectType ZOOKEEPER

2014-06-27 11:05:50,192 INFO [main:mortbay.log@67] Registered SubjectType ZOOKEEPER-SERVER

2014-06-27 11:05:50,192 INFO [main:mortbay.log@67] Registered SubjectType FLUME

2014-06-27 11:05:50,192 INFO [main:mortbay.log@67] Registered SubjectType FLUME-AGENT

2014-06-27 11:05:50,193 INFO [main:mortbay.log@67] Registered SubjectType IMPALA

2014-06-27 11:05:50,193 INFO [main:mortbay.log@67] Registered SubjectType IMPALA-IMPALAD

2014-06-27 11:05:50,194 INFO [main:mortbay.log@67] Registered SubjectType IMPALA-STATESTORE

2014-06-27 11:05:50,195 INFO [main:mortbay.log@67] Registered SubjectType IMPALA-CATALOGSERVER

2014-06-27 11:05:50,195 INFO [main:mortbay.log@67] Registered SubjectType IMPALA-LLAMA

2014-06-27 11:05:50,196 INFO [main:mortbay.log@67] Registered SubjectType YARN

2014-06-27 11:05:50,197 INFO [main:mortbay.log@67] Registered SubjectType YARN-RESOURCEMANAGER

2014-06-27 11:05:50,197 INFO [main:mortbay.log@67] Registered SubjectType YARN-NODEMANAGER

2014-06-27 11:05:50,198 INFO [main:mortbay.log@67] Registered SubjectType YARN-JOBHISTORY

2014-06-27 11:05:50,199 INFO [main:mortbay.log@67] Registered SubjectType OOZIE

2014-06-27 11:05:50,199 INFO [main:mortbay.log@67] Registered SubjectType OOZIE-OOZIE_SERVER

2014-06-27 11:05:50,200 INFO [main:mortbay.log@67] Registered SubjectType HIVE

2014-06-27 11:05:50,200 INFO [main:mortbay.log@67] Registered SubjectType HIVE-HIVEMETASTORE

2014-06-27 11:05:50,201 INFO [main:mortbay.log@67] Registered SubjectType HIVE-WEBHCAT

2014-06-27 11:05:50,202 INFO [main:mortbay.log@67] Registered SubjectType HIVE-HIVESERVER2

2014-06-27 11:05:50,202 INFO [main:mortbay.log@67] Registered SubjectType HUE

2014-06-27 11:05:50,203 INFO [main:mortbay.log@67] Registered SubjectType HUE-HUE_SERVER

2014-06-27 11:05:50,203 INFO [main:mortbay.log@67] Registered SubjectType HUE-BEESWAX_SERVER

2014-06-27 11:05:50,204 INFO [main:mortbay.log@67] Registered SubjectType HUE-KT_RENEWER

2014-06-27 11:05:50,204 INFO [main:mortbay.log@67] Registered SubjectType SQOOP

2014-06-27 11:05:50,205 INFO [main:mortbay.log@67] Registered SubjectType SQOOP-SQOOP_SERVER

2014-06-27 11:05:50,205 INFO [main:mortbay.log@67] Registered SubjectType SOLR

2014-06-27 11:05:50,206 INFO [main:mortbay.log@67] Registered SubjectType SOLR-SOLR_SERVER

2014-06-27 11:05:50,206 INFO [main:mortbay.log@67] Registered SubjectType KS_INDEXER

2014-06-27 11:05:50,207 INFO [main:mortbay.log@67] Registered SubjectType KS_INDEXER-HBASE_INDEXER

2014-06-27 11:05:50,279 INFO [main:support.ClassPathXmlApplicationContext@456] Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@53c5472f: startup date [Fri Jun 27 11:05:50 EDT 2014]; root of context hierarchy

2014-06-27 11:05:50,354 INFO [main:xml.XmlBeanDefinitionReader@315] Loading XML bean definitions from class path resource [webapp/WEB-INF/spring/beanRefFactory.xml]

2014-06-27 11:05:50,814 INFO [main:support.DefaultListableBeanFactory@557] Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@66652f4d: defining beans [bootstrapContext,rootContext]; root of factory hierarchy

2014-06-27 11:05:50,858 INFO [main:support.GenericApplicationContext@456] Refreshing org.springframework.context.support.GenericApplicationContext@75ea6b61: startup date [Fri Jun 27 11:05:50 EDT 2014]; root of context hierarchy

2014-06-27 11:05:50,863 INFO [main:support.DefaultListableBeanFactory@557] Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4bdc0b6b: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy

2014-06-27 11:05:51,001 INFO [main:enterprise.CommonMain@105] Reading database properties from /etc/cloudera-scm-server/db.properties

2014-06-27 11:05:51,009 INFO [main:enterprise.CommonMain@148] Statistics not enabled, c3p0 JMX disabled

2014-06-27 11:05:51,230 INFO [main:common.Version@37] HCANN000001: Hibernate Commons Annotations {4.0.2.Final}

2014-06-27 11:05:51,239 INFO [main:hibernate.Version@41] HHH000412: Hibernate Core {4.2.2.Final}

2014-06-27 11:05:51,243 INFO [main:cfg.Environment@239] HHH000206: hibernate.properties not found

2014-06-27 11:05:51,246 INFO [main:cfg.Environment@342] HHH000021: Bytecode provider name : javassist

2014-06-27 11:05:51,725 INFO [main:cfg.Configuration@728] HHH000221: Reading mappings from resource: cmf.hbm.xml

2014-06-27 11:05:52,205 INFO [main:internal.ConnectionProviderInitiator@188] HHH000130: Instantiating explicit connection provider: org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider

2014-06-27 11:05:52,207 INFO [main:internal.C3P0ConnectionProvider@128] HHH010002: C3P0 using driver: org.postgresql.Driver at URL: jdbc:postgresql://localhost:7432/scm

2014-06-27 11:05:52,207 INFO [main:internal.C3P0ConnectionProvider@129] HHH000046: Connection properties: {user=scm, password=****, autocommit=true, release_mode=auto}

2014-06-27 11:05:52,207 INFO [main:internal.C3P0ConnectionProvider@132] HHH000006: Autocommit mode: true

2014-06-27 11:05:52,239 INFO [main:log.MLog@80] MLog clients using log4j logging.

2014-06-27 11:05:52,401 INFO [main:c3p0.C3P0Registry@204] Initializing c3p0-0.9.1.1 [built 15-March-2007 01:32:31; debug? true; trace: 10]

2014-06-27 11:05:52,491 INFO [main:internal.C3P0ConnectionProvider@214] HHH000149: JDBC isolation level: READ_COMMITTED

2014-06-27 11:05:52,565 INFO [main:impl.AbstractPoolBackedDataSource@462] Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@c8dc2cad [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@81cf1e07 [ acquireIncrement -> 3, acquireRetryAttempts -> 5, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 20000, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2y2far921pk3yje19mjsi0|37af5cd6, idleConnectionTestPeriod -> 300, initialPoolSize -> 3, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 2500, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@5dab41fd [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 2y2far921pk3yje19mjsi0|7ecc938c, jdbcUrl -> jdbc:postgresql://localhost:7432/scm, properties -> {user=******, password=******, autocommit=true, release_mode=auto} ], preferredTestQuery -> null, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 2y2far921pk3yje19mjsi0|792c68e3, numHelperThreads -> 3 ]

2014-06-27 11:05:56,657 WARN [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@73a1eeaa -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:

org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136)

at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)

at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)

at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)

at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)

at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)

at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)

at org.postgresql.Driver.makeConnection(Driver.java:393)

at org.postgresql.Driver.connect(Driver.java:267)

at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)

at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)

at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)

at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)

at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)

at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Caused by: java.net.ConnectException: Connection refused

6 REPLIES 6

Re: Can't start Cloudera Manager or Hadoop Services after hard restart

Explorer

Issue was with the database permissions. data folder is made global and when I reverted back to 0700 permissions, it worked!!

Re: Can't start Cloudera Manager or Hadoop Services after hard restart

Explorer

Hi,

I am stuck at the same issue. May I know how you fixed it.

Thanks,

Akhil

Re: Can't start Cloudera Manager or Hadoop Services after hard restart

Explorer

My Error log for the CM agent server is something like this:

 

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.sql.SQLException: Schema version table SCHEMA_VERSION exists but contains no rows.
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 17 more
Caused by: java.sql.SQLException: Schema version table SCHEMA_VERSION exists but contains no rows.
at com.cloudera.enterprise.dbutil.DbUtil.getSchemaVersion(DbUtil.java:232)
at com.cloudera.enterprise.dbutil.DbUtil$1SchemaVersionWork.execute(DbUtil.java:171)
at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:54)
at org.hibernate.internal.SessionImpl$2.accept(SessionImpl.java:1982)
at org.hibernate.internal.SessionImpl$2.accept(SessionImpl.java:1979)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:284)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2000)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:1986)
at com.cloudera.enterprise.dbutil.DbUtil.getSchemaVersion(DbUtil.java:183)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.checkVersionDoFail(EntityManagerFactoryBean.java:158)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:104)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:50)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
... 22 more
2014-09-28 16:41:00,319 INFO [main:mortbay.log@67] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2014-09-28 16:41:00,321 INFO [main:mortbay.log@67] Registered SubjectType HDFS
2014-09-28 16:41:00,326 INFO [main:mortbay.log@67] Registered SubjectType HDFS-NAMENODE
2014-09-28 16:41:00,329 INFO [main:mortbay.log@67] Registered SubjectType HDFS-DATANODE
2014-09-28 16:41:00,329 INFO [main:mortbay.log@67] Registered SubjectType HDFS-SECONDARYNAMENODE
2014-09-28 16:41:00,330 INFO [main:mortbay.log@67] Registered SubjectType HDFS-JOURNALNODE
2014-09-28 16:41:00,330 INFO [main:mortbay.log@67] Registered SubjectType HDFS-HTTPFS
2014-09-28 16:41:00,330 INFO [main:mortbay.log@67] Registered SubjectType HDFS-FAILOVERCONTROLLER
2014-09-28 16:41:00,331 INFO [main:mortbay.log@67] Registered SubjectType HDFS-NFSGATEWAY
2014-09-28 16:41:00,331 INFO [main:mortbay.log@67] Registered SubjectType HBASE
2014-09-28 16:41:00,331 INFO [main:mortbay.log@67] Registered SubjectType HBASE-MASTER
2014-09-28 16:41:00,332 INFO [main:mortbay.log@67] Registered SubjectType HBASE-REGIONSERVER
2014-09-28 16:41:00,332 INFO [main:mortbay.log@67] Registered SubjectType HBASE-HBASETHRIFTSERVER
2014-09-28 16:41:00,332 INFO [main:mortbay.log@67] Registered SubjectType HBASE-HBASERESTSERVER
2014-09-28 16:41:00,333 INFO [main:mortbay.log@67] Registered SubjectType MAPREDUCE
2014-09-28 16:41:00,333 INFO [main:mortbay.log@67] Registered SubjectType MAPREDUCE-JOBTRACKER
2014-09-28 16:41:00,335 INFO [main:mortbay.log@67] Registered SubjectType MAPREDUCE-TASKTRACKER
2014-09-28 16:41:00,336 INFO [main:mortbay.log@67] Registered SubjectType MAPREDUCE-FAILOVERCONTROLLER
2014-09-28 16:41:00,337 INFO [main:mortbay.log@67] Registered SubjectType MGMT
2014-09-28 16:41:00,337 INFO [main:mortbay.log@67] Registered SubjectType MGMT-ACTIVITYMONITOR
2014-09-28 16:41:00,337 INFO [main:mortbay.log@67] Registered SubjectType MGMT-SERVICEMONITOR
2014-09-28 16:41:00,338 INFO [main:mortbay.log@67] Registered SubjectType MGMT-HOSTMONITOR
2014-09-28 16:41:00,338 INFO [main:mortbay.log@67] Registered SubjectType MGMT-EVENTSERVER
2014-09-28 16:41:00,339 INFO [main:mortbay.log@67] Registered SubjectType MGMT-ALERTPUBLISHER
2014-09-28 16:41:00,339 INFO [main:mortbay.log@67] Registered SubjectType MGMT-REPORTSMANAGER
2014-09-28 16:41:00,339 INFO [main:mortbay.log@67] Registered SubjectType MGMT-NAVIGATOR
2014-09-28 16:41:00,340 INFO [main:mortbay.log@67] Registered SubjectType MGMT-NAVIGATORMETASERVER
2014-09-28 16:41:00,340 INFO [main:mortbay.log@67] Registered SubjectType HOST
2014-09-28 16:41:00,340 INFO [main:mortbay.log@67] Registered SubjectType ZOOKEEPER
2014-09-28 16:41:00,340 INFO [main:mortbay.log@67] Registered SubjectType ZOOKEEPER-SERVER
2014-09-28 16:41:00,340 INFO [main:mortbay.log@67] Registered SubjectType FLUME
2014-09-28 16:41:00,341 INFO [main:mortbay.log@67] Registered SubjectType FLUME-AGENT
2014-09-28 16:41:00,341 INFO [main:mortbay.log@67] Registered SubjectType IMPALA
2014-09-28 16:41:00,341 INFO [main:mortbay.log@67] Registered SubjectType IMPALA-IMPALAD
2014-09-28 16:41:00,342 INFO [main:mortbay.log@67] Registered SubjectType IMPALA-STATESTORE
2014-09-28 16:41:00,342 INFO [main:mortbay.log@67] Registered SubjectType IMPALA-CATALOGSERVER
2014-09-28 16:41:00,342 INFO [main:mortbay.log@67] Registered SubjectType IMPALA-LLAMA
2014-09-28 16:41:00,342 INFO [main:mortbay.log@67] Registered SubjectType YARN
2014-09-28 16:41:00,343 INFO [main:mortbay.log@67] Registered SubjectType YARN-RESOURCEMANAGER
2014-09-28 16:41:00,343 INFO [main:mortbay.log@67] Registered SubjectType YARN-NODEMANAGER
2014-09-28 16:41:00,344 INFO [main:mortbay.log@67] Registered SubjectType YARN-JOBHISTORY
2014-09-28 16:41:00,344 INFO [main:mortbay.log@67] Registered SubjectType OOZIE
2014-09-28 16:41:00,344 INFO [main:mortbay.log@67] Registered SubjectType OOZIE-OOZIE_SERVER
2014-09-28 16:41:00,344 INFO [main:mortbay.log@67] Registered SubjectType HIVE
2014-09-28 16:41:00,345 INFO [main:mortbay.log@67] Registered SubjectType HIVE-HIVEMETASTORE
2014-09-28 16:41:00,345 INFO [main:mortbay.log@67] Registered SubjectType HIVE-WEBHCAT
2014-09-28 16:41:00,345 INFO [main:mortbay.log@67] Registered SubjectType HIVE-HIVESERVER2
2014-09-28 16:41:00,346 INFO [main:mortbay.log@67] Registered SubjectType HUE
2014-09-28 16:41:00,346 INFO [main:mortbay.log@67] Registered SubjectType HUE-HUE_SERVER
2014-09-28 16:41:00,346 INFO [main:mortbay.log@67] Registered SubjectType HUE-BEESWAX_SERVER
2014-09-28 16:41:00,347 INFO [main:mortbay.log@67] Registered SubjectType HUE-KT_RENEWER
2014-09-28 16:41:00,347 INFO [main:mortbay.log@67] Registered SubjectType SENTRY
2014-09-28 16:41:00,347 INFO [main:mortbay.log@67] Registered SubjectType SENTRY-SENTRY_SERVER
2014-09-28 16:41:00,347 INFO [main:mortbay.log@67] Registered SubjectType SQOOP
2014-09-28 16:41:00,348 INFO [main:mortbay.log@67] Registered SubjectType SQOOP-SQOOP_SERVER
2014-09-28 16:41:00,348 INFO [main:mortbay.log@67] Registered SubjectType SOLR
2014-09-28 16:41:00,348 INFO [main:mortbay.log@67] Registered SubjectType SOLR-SOLR_SERVER
2014-09-28 16:41:00,348 INFO [main:mortbay.log@67] Registered SubjectType KS_INDEXER
2014-09-28 16:41:00,349 INFO [main:mortbay.log@67] Registered SubjectType KS_INDEXER-HBASE_INDEXER
2014-09-28 16:41:00,409 INFO [main:support.ClassPathXmlApplicationContext@456] Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@7bfbfeae: startup date [Sun Sep 28 16:41:00 EDT 2014]; root of context hierarchy
2014-09-28 16:41:00,476 INFO [main:xml.XmlBeanDefinitionReader@315] Loading XML bean definitions from class path resource [webapp/WEB-INF/spring/beanRefFactory.xml]
2014-09-28 16:41:00,646 INFO [main:support.DefaultListableBeanFactory@557] Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@30a491e9: defining beans [bootstrapContext,rootContext]; root of factory hierarchy
2014-09-28 16:41:00,681 INFO [main:support.GenericApplicationContext@456] Refreshing org.springframework.context.support.GenericApplicationContext@7525c6bd: startup date [Sun Sep 28 16:41:00 EDT 2014]; root of context hierarchy
2014-09-28 16:41:00,683 INFO [main:support.DefaultListableBeanFactory@557] Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@715dcdeb: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy
2014-09-28 16:41:00,792 INFO [main:enterprise.CommonMain@105] Reading database properties from /etc/cloudera-scm-server/db.properties
2014-09-28 16:41:00,797 INFO [main:enterprise.CommonMain@166] Statistics not enabled, c3p0 JMX disabled
2014-09-28 16:41:00,982 INFO [main:common.Version@37] HCANN000001: Hibernate Commons Annotations {4.0.2.Final}
2014-09-28 16:41:00,990 INFO [main:hibernate.Version@41] HHH000412: Hibernate Core {4.2.2.Final}
2014-09-28 16:41:00,993 INFO [main:cfg.Environment@239] HHH000206: hibernate.properties not found
2014-09-28 16:41:00,994 INFO [main:cfg.Environment@342] HHH000021: Bytecode provider name : javassist
2014-09-28 16:41:01,377 INFO [main:cfg.Configuration@728] HHH000221: Reading mappings from resource: cmf.hbm.xml
2014-09-28 16:41:01,670 INFO [main:internal.ConnectionProviderInitiator@188] HHH000130: Instantiating explicit connection provider: org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider
2014-09-28 16:41:01,671 INFO [main:internal.C3P0ConnectionProvider@128] HHH010002: C3P0 using driver: com.mysql.jdbc.Driver at URL: jdbc:mysql://localhost/cman?useUnicode=true&characterEncoding=UTF-8
2014-09-28 16:41:01,672 INFO [main:internal.C3P0ConnectionProvider@129] HHH000046: Connection properties: {user=cman, password=****, autocommit=true, release_mode=auto}
2014-09-28 16:41:01,672 INFO [main:internal.C3P0ConnectionProvider@132] HHH000006: Autocommit mode: true
2014-09-28 16:41:01,750 INFO [main:log.MLog@80] MLog clients using log4j logging.
2014-09-28 16:41:01,875 INFO [main:c3p0.C3P0Registry@204] Initializing c3p0-0.9.1.1 [built 15-March-2007 01:32:31; debug? true; trace: 10]
2014-09-28 16:41:01,929 INFO [main:internal.C3P0ConnectionProvider@214] HHH000149: JDBC isolation level: READ_COMMITTED
2014-09-28 16:41:01,975 INFO [main:impl.AbstractPoolBackedDataSource@462] Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@85183d77 [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@21731cbd [ acquireIncrement -> 3, acquireRetryAttempts -> 5, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 20000, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1br1c1q941gpt36p1o3apug|6b0f6d29, idleConnectionTestPeriod -> 300, initialPoolSize -> 3, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 2500, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@8da4ce69 [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 1br1c1q941gpt36p1o3apug|21ab613c, jdbcUrl -> jdbc:mysql://localhost/cman?useUnicode=true&characterEncoding=UTF-8, properties -> {user=******, password=******, autocommit=true, release_mode=auto} ], preferredTestQuery -> null, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 1br1c1q941gpt36p1o3apug|4d338bcf, numHelperThreads -> 3 ]
2014-09-28 16:41:02,189 INFO [main:dialect.Dialect@130] HHH000400: Using dialect: org.hibernate.dialect.MySQL5InnoDBDialect
2014-09-28 16:41:02,199 INFO [main:internal.LobCreatorBuilder@120] HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException
2014-09-28 16:41:02,350 INFO [main:internal.TransactionFactoryInitiator@73] HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory
2014-09-28 16:41:02,355 INFO [main:ast.ASTQueryTranslatorFactory@48] HHH000397: Using ASTQueryTranslatorFactory
2014-09-28 16:41:02,480 INFO [main:spi.UpdateTimestampsCache@61] HHH000250: Starting update timestamps cache at region: org.hibernate.cache.spi.UpdateTimestampsCache
2014-09-28 16:41:02,483 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.spi.UpdateTimestampsCache]; using defaults.
2014-09-28 16:41:02,485 INFO [main:internal.StandardQueryCache@86] HHH000248: Starting query cache at region: org.hibernate.cache.internal.StandardQueryCache
2014-09-28 16:41:02,486 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.internal.StandardQueryCache]; using defaults.
2014-09-28 16:41:02,516 INFO [main:util.Version@27] HV000001: Hibernate Validator 5.0.1.Final
2014-09-28 16:41:03,189 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbHost]; using defaults.
2014-09-28 16:41:03,308 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCredential]; using defaults.
2014-09-28 16:41:03,312 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbService]; using defaults.
2014-09-28 16:41:03,351 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbRoleConfigGroup]; using defaults.
2014-09-28 16:41:03,359 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCmPeer]; using defaults.
2014-09-28 16:41:03,363 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCluster]; using defaults.
2014-09-28 16:41:03,371 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbConfigContainer]; using defaults.
2014-09-28 16:41:03,393 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbHostTemplate]; using defaults.
2014-09-28 16:41:03,401 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbClientConfig]; using defaults.
2014-09-28 16:41:03,408 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbUser]; using defaults.
2014-09-28 16:41:03,413 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbGlobalSetting]; using defaults.
2014-09-28 16:41:03,416 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbRevision]; using defaults.
2014-09-28 16:41:03,421 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbUserSetting]; using defaults.
2014-09-28 16:41:03,425 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCommandSchedule]; using defaults.
2014-09-28 16:41:03,441 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbProcess]; using defaults.
2014-09-28 16:41:03,621 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbUserRole]; using defaults.
2014-09-28 16:41:03,624 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbParcel]; using defaults.
2014-09-28 16:41:03,632 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbRole]; using defaults.
2014-09-28 16:41:03,641 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbRelease]; using defaults.
2014-09-28 16:41:03,646 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbConfig]; using defaults.
2014-09-28 16:41:03,692 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbService.configsForDb]; using defaults.
2014-09-28 16:41:03,695 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCluster.activatedReleases]; using defaults.
2014-09-28 16:41:03,698 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbHostTemplate.roleConfigGroups]; using defaults.
2014-09-28 16:41:03,699 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbRoleConfigGroup.roles]; using defaults.
2014-09-28 16:41:03,700 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbHost.roles]; using defaults.
2014-09-28 16:41:03,701 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbService.roleConfigGroups]; using defaults.
2014-09-28 16:41:03,702 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbConfigContainer.configsForDb]; using defaults.
2014-09-28 16:41:03,703 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbProcess.parcels]; using defaults.
2014-09-28 16:41:03,703 WARN [main:ehcache.AbstractEhcacheRegionFactory@180] HHH020003: Could not find a specific ehcache configuration for cache named [com.cloudera.cmf.model.DbCluster.hostTemplates]; using defaults.
2014-09-28 16:41:04,422 INFO [main:enterprise.CommonMain@261] Statistics not enabled, JMX will not be registered
2014-09-28 16:41:04,543 INFO [main:bootstrap.EntityManagerFactoryBean@135] MYSQL database engine and table mapping: {InnoDB=[SCHEMA_VERSION]}
2014-09-28 16:41:04,551 INFO [main:dbutil.DbUtil@219] Schema version table already exists.
2014-09-28 16:41:04,552 ERROR [main:dbutil.DbUtil@231] Schema version table SCHEMA_VERSION exists but contains no rows.
2014-09-28 16:41:04,553 INFO [main:support.DefaultListableBeanFactory@422] Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@715dcdeb: defining beans [commandLineConfigurationBean,entityManagerFactoryBean,com.cloudera.server.cmf.TrialState,com.cloudera.server.cmf.TrialManager,com.cloudera.cmf.crypto.LicenseLoader]; root of factory hierarchy
2014-09-28 16:41:04,553 ERROR [main:cmf.Main@204] Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.sql.SQLException: Schema version table SCHEMA_VERSION exists but contains no rows.
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:362)
at com.cloudera.server.cmf.Main.<init>(Main.java:216)
at com.cloudera.server.cmf.Main.main(Main.java:199)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.sql.SQLException: Schema version table SCHEMA_VERSION exists but contains no rows.
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 17 more
Caused by: java.sql.SQLException: Schema version table SCHEMA_VERSION exists but contains no rows.
at com.cloudera.enterprise.dbutil.DbUtil.getSchemaVersion(DbUtil.java:232)
at com.cloudera.enterprise.dbutil.DbUtil$1SchemaVersionWork.execute(DbUtil.java:171)
at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:54)
at org.hibernate.internal.SessionImpl$2.accept(SessionImpl.java:1982)
at org.hibernate.internal.SessionImpl$2.accept(SessionImpl.java:1979)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:284)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2000)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:1986)
at com.cloudera.enterprise.dbutil.DbUtil.getSchemaVersion(DbUtil.java:183)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.checkVersionDoFail(EntityManagerFactoryBean.java:158)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:104)
at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:50)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142)
... 22 more

Re: Can't start Cloudera Manager or Hadoop Services after hard restart

New Contributor

drop table SCHEMA_VERSION and restart. This worked for me.

Re: Can't start Cloudera Manager or Hadoop Services after hard restart

Explorer

Sure. I will try that the next time I encounter this issue. I have used a workaround at the moment.

 

Re: Can't start Cloudera Manager or Hadoop Services after hard restart

Hi ,

 

What does this mean - data folder is made global ?  do we need to change directory permission to root:root or mysql:mysql.

 

I got stuck up with same error

 

Thanks

Kishore

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here