Cloudera manager JDBC connection errors

[#1]

Hi, I'm using

Cloudera Express 5.4.1 (#197 built by jenkins on 20150509-0042 git: 003e06d761f80834d39c3a42431a266f0aaee736)

The cluster (hdfs, yarn, etc) is still a fully operational battle station but CM is borked, the error below pops up repeatedly.

Any help on how to proceed debugging this would be great,

Thanks,

James

Server Error

A server error has occurred. Send the following information to Cloudera.

Path: http://192.168.0.86:7180/cmf/home

Version: Cloudera Express 5.4.1 (#197 built by jenkins on 20150509-0042 git: 003e06d761f80834d39c3a42431a266f0aaee736)

javax.persistence.PersistenceException:org.hibernate.exception.GenericJDBCException: Could not open connection
at AbstractEntityManagerImpl.java line 1387
in org.hibernate.ejb.AbstractEntityManagerImpl convert()

Stack Trace:

1. AbstractEntityManagerImpl.java line 1387
   in org.hibernate.ejb.AbstractEntityManagerImpl convert()
2. AbstractEntityManagerImpl.java line 1310
   in org.hibernate.ejb.AbstractEntityManagerImpl convert()
3. AbstractEntityManagerImpl.java line 1397
   in org.hibernate.ejb.AbstractEntityManagerImpl throwPersistenceException()
4. TransactionImpl.java line 62
   in org.hibernate.ejb.TransactionImpl begin()
5. AbstractWrappedEntityManager.java line 85
   in com.cloudera.enterprise.AbstractWrappedEntityManager beginForRollbackAndReadonly()
6. CmfEntityManager.java line 335
   in com.cloudera.cmf.persist.CmfEntityManager beginForRollbackAndReadonly()
7. AggregateStatusController.java line 182
   in com.cloudera.server.web.cmf.AggregateStatusController landingPage()
8. line -1
   in sun.reflect.GeneratedMethodAccessor1639 invoke()
9. DelegatingMethodAccessorImpl.java line 43
   in sun.reflect.DelegatingMethodAccessorImpl invoke()
10. Method.java line 497
    in java.lang.reflect.Method invoke()
11. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
12. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
13. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
14. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
15. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
16. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
17. FrameworkServlet.java line 574
    in org.springframework.web.servlet.FrameworkServlet doGet()
18. HttpServlet.java line 575
    in javax.servlet.http.HttpServlet service()
19. HttpServlet.java line 668
    in javax.servlet.http.HttpServlet service()
20. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
21. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
22. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
23. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
24. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
25. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
26. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
27. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
28. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
29. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
30. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
31. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
32. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
33. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
34. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
35. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
36. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
37. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
38. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
39. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
40. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
41. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
42. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
43. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
44. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
45. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
46. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
47. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
48. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
50. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
51. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
52. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
53. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
54. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
55. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
56. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
57. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
58. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
59. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
60. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
61. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
62. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
63. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
64. ContextHandler.java line 767
    in org.mortbay.jetty.handler.ContextHandler handle()
65. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
66. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
67. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
68. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
69. Server.java line 326
    in org.mortbay.jetty.Server handle()
70. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
71. HttpConnection.java line 928
    in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
72. HttpParser.java line 549
    in org.mortbay.jetty.HttpParser parseNext()
73. HttpParser.java line 212
    in org.mortbay.jetty.HttpParser parseAvailable()
74. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
75. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
76. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: org.hibernate.exception.GenericJDBCException:Could not open connection
at StandardSQLExceptionConverter.java line 54
in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()

Stack Trace:

1. StandardSQLExceptionConverter.java line 54
   in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()
2. SqlExceptionHelper.java line 125
   in org.hibernate.engine.jdbc.spi.SqlExceptionHelper convert()
3. SqlExceptionHelper.java line 110
   in org.hibernate.engine.jdbc.spi.SqlExceptionHelper convert()
4. LogicalConnectionImpl.java line 221
   in org.hibernate.engine.jdbc.internal.LogicalConnectionImpl obtainConnection()
5. LogicalConnectionImpl.java line 157
   in org.hibernate.engine.jdbc.internal.LogicalConnectionImpl getConnection()
6. JdbcTransaction.java line 67
   in org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction doBegin()
7. AbstractTransactionImpl.java line 160
   in org.hibernate.engine.transaction.spi.AbstractTransactionImpl begin()
8. SessionImpl.java line 1426
   in org.hibernate.internal.SessionImpl beginTransaction()
9. TransactionImpl.java line 59
   in org.hibernate.ejb.TransactionImpl begin()
10. AbstractWrappedEntityManager.java line 85
    in com.cloudera.enterprise.AbstractWrappedEntityManager beginForRollbackAndReadonly()
11. CmfEntityManager.java line 335
    in com.cloudera.cmf.persist.CmfEntityManager beginForRollbackAndReadonly()
12. AggregateStatusController.java line 182
    in com.cloudera.server.web.cmf.AggregateStatusController landingPage()
13. line -1
    in sun.reflect.GeneratedMethodAccessor1639 invoke()
14. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
15. Method.java line 497
    in java.lang.reflect.Method invoke()
16. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
17. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
18. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
19. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
20. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
21. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
22. FrameworkServlet.java line 574
    in org.springframework.web.servlet.FrameworkServlet doGet()
23. HttpServlet.java line 575
    in javax.servlet.http.HttpServlet service()
24. HttpServlet.java line 668
    in javax.servlet.http.HttpServlet service()
25. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
26. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
27. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
28. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
29. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
30. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
31. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
32. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
33. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
34. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
35. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
36. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
37. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
38. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
39. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
40. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
41. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
42. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
43. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
44. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
45. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
46. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
47. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
48. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
49. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
50. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
51. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
52. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
53. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
54. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
55. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
56. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
57. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
58. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
59. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
60. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
61. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
62. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
63. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
64. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
65. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
66. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
67. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
68. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
69. ContextHandler.java line 767
    in org.mortbay.jetty.handler.ContextHandler handle()
70. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
71. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
72. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
73. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
74. Server.java line 326
    in org.mortbay.jetty.Server handle()
75. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
76. HttpConnection.java line 928
    in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
77. HttpParser.java line 549
    in org.mortbay.jetty.HttpParser parseNext()
78. HttpParser.java line 212
    in org.mortbay.jetty.HttpParser parseAvailable()
79. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
80. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
81. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: java.sql.SQLException:Connections could not be acquired from the underlying database!
at SqlUtils.java line 106
in com.mchange.v2.sql.SqlUtils toSQLException()

Stack Trace:

1. SqlUtils.java line 106
   in com.mchange.v2.sql.SqlUtils toSQLException()
2. C3P0PooledConnectionPool.java line 529
   in com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool checkoutPooledConnection()
3. AbstractPoolBackedDataSource.java line 128
   in com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getConnection()
4. C3P0ConnectionProvider.java line 84
   in org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider getConnection()
5. AbstractSessionImpl.java line 292
   in org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess obtainConnection()
6. LogicalConnectionImpl.java line 214
   in org.hibernate.engine.jdbc.internal.LogicalConnectionImpl obtainConnection()
7. LogicalConnectionImpl.java line 157
   in org.hibernate.engine.jdbc.internal.LogicalConnectionImpl getConnection()
8. JdbcTransaction.java line 67
   in org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction doBegin()
9. AbstractTransactionImpl.java line 160
   in org.hibernate.engine.transaction.spi.AbstractTransactionImpl begin()
10. SessionImpl.java line 1426
    in org.hibernate.internal.SessionImpl beginTransaction()
11. TransactionImpl.java line 59
    in org.hibernate.ejb.TransactionImpl begin()
12. AbstractWrappedEntityManager.java line 85
    in com.cloudera.enterprise.AbstractWrappedEntityManager beginForRollbackAndReadonly()
13. CmfEntityManager.java line 335
    in com.cloudera.cmf.persist.CmfEntityManager beginForRollbackAndReadonly()
14. AggregateStatusController.java line 182
    in com.cloudera.server.web.cmf.AggregateStatusController landingPage()
15. line -1
    in sun.reflect.GeneratedMethodAccessor1639 invoke()
16. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
17. Method.java line 497
    in java.lang.reflect.Method invoke()
18. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
19. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
20. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
21. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
22. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
23. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
24. FrameworkServlet.java line 574
    in org.springframework.web.servlet.FrameworkServlet doGet()
25. HttpServlet.java line 575
    in javax.servlet.http.HttpServlet service()
26. HttpServlet.java line 668
    in javax.servlet.http.HttpServlet service()
27. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
28. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
29. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
30. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
31. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
32. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
33. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
34. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
35. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
36. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
37. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
38. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
39. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
40. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
41. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
42. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
43. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
44. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
45. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
46. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
47. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
48. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
49. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
50. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
51. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
52. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
53. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
54. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
55. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
56. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
57. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
58. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
59. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
60. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
61. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
62. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
63. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
64. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
65. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
66. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
67. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
68. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
69. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
70. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
71. ContextHandler.java line 767
    in org.mortbay.jetty.handler.ContextHandler handle()
72. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
73. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
74. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
75. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
76. Server.java line 326
    in org.mortbay.jetty.Server handle()
77. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
78. HttpConnection.java line 928
    in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
79. HttpParser.java line 549
    in org.mortbay.jetty.HttpParser parseNext()
80. HttpParser.java line 212
    in org.mortbay.jetty.HttpParser parseAvailable()
81. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
82. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
83. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException:A ResourcePool could not acquire a resource from its primary factory or source.
at BasicResourcePool.java line 1319
in com.mchange.v2.resourcepool.BasicResourcePool awaitAvailable()

Stack Trace:

1. BasicResourcePool.java line 1319
   in com.mchange.v2.resourcepool.BasicResourcePool awaitAvailable()
2. BasicResourcePool.java line 557
   in com.mchange.v2.resourcepool.BasicResourcePool prelimCheckoutResource()
3. BasicResourcePool.java line 477
   in com.mchange.v2.resourcepool.BasicResourcePool checkoutResource()
4. C3P0PooledConnectionPool.java line 525
   in com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool checkoutPooledConnection()
5. AbstractPoolBackedDataSource.java line 128
   in com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getConnection()
6. C3P0ConnectionProvider.java line 84
   in org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider getConnection()
7. AbstractSessionImpl.java line 292
   in org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess obtainConnection()
8. LogicalConnectionImpl.java line 214
   in org.hibernate.engine.jdbc.internal.LogicalConnectionImpl obtainConnection()
9. LogicalConnectionImpl.java line 157
   in org.hibernate.engine.jdbc.internal.LogicalConnectionImpl getConnection()
10. JdbcTransaction.java line 67
    in org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction doBegin()
11. AbstractTransactionImpl.java line 160
    in org.hibernate.engine.transaction.spi.AbstractTransactionImpl begin()
12. SessionImpl.java line 1426
    in org.hibernate.internal.SessionImpl beginTransaction()
13. TransactionImpl.java line 59
    in org.hibernate.ejb.TransactionImpl begin()
14. AbstractWrappedEntityManager.java line 85
    in com.cloudera.enterprise.AbstractWrappedEntityManager beginForRollbackAndReadonly()
15. CmfEntityManager.java line 335
    in com.cloudera.cmf.persist.CmfEntityManager beginForRollbackAndReadonly()
16. AggregateStatusController.java line 182
    in com.cloudera.server.web.cmf.AggregateStatusController landingPage()
17. line -1
    in sun.reflect.GeneratedMethodAccessor1639 invoke()
18. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
19. Method.java line 497
    in java.lang.reflect.Method invoke()
20. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
21. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
22. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
23. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
24. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
25. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
26. FrameworkServlet.java line 574
    in org.springframework.web.servlet.FrameworkServlet doGet()
27. HttpServlet.java line 575
    in javax.servlet.http.HttpServlet service()
28. HttpServlet.java line 668
    in javax.servlet.http.HttpServlet service()
29. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
30. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
31. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
32. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
33. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
34. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
35. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
36. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
37. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
38. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
39. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
40. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
41. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
42. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
43. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
44. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
45. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
46. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
47. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
48. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
49. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
50. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
51. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
52. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
53. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
54. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
55. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
56. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
57. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
58. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
59. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
60. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
61. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
62. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
63. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
64. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
65. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
66. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
67. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
68. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
69. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
70. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
71. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
72. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
73. ContextHandler.java line 767
    in org.mortbay.jetty.handler.ContextHandler handle()
74. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
75. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
76. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
77. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
78. Server.java line 326
    in org.mortbay.jetty.Server handle()
79. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
80. HttpConnection.java line 928
    in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
81. HttpParser.java line 549
    in org.mortbay.jetty.HttpParser parseNext()
82. HttpParser.java line 212
    in org.mortbay.jetty.HttpParser parseAvailable()
83. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
84. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
85. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

[#2 · Accepted solution]

Sounds like CM can't talk to its own database. Verify that the database is up and running and the user that CM is using can talk to it. See connection info in /etc/cloudera-scm-server/db.properties.

Thanks,
Darren

[#3]

Thanks for the pointer. I get this:

cloud-user@cdh-node0:~$ psql -d scm -U scm -W

Password for user scm: 

psql: FATAL:  Peer authentication failed for user "scm"

I entered the password from the file, am I doing something wrong there or is that showing up the problem do you think?

Cheers,

James

[#4]

I had some iptables rules that were breaking things, flushed them and it's OK now, thanks for the help Darren.

[#5]

Hi,

I am facing the same exception after restarting the clouder server

Mar 18 09:27:36 xxx.example.com cm-server[8420]: 09:27:36.744 [ScmActive-0] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Lock wait timeo...nsaction

I have checked the database and its up and running.

Could you please provide more insight on 'iptables'

Thanks,

Deiveegan

[#6]

iptables is a linux firewall, in my case it was blocking the traffic needed for the database connection