Server error has occurred

[#1]

Hello ,

One of my datanode stopped working , By mistake /etc/init.d/network stop is been used . After then i resolved the issue and tried opening my CM . Then I am seeing this error

A server error has occurred. Send the following information to Cloudera.

Path: http://127.0.0.1:7180/cmf/services/landingPageStatusContent

Version: Cloudera Enterprise Data Hub Edition Trial 5.0.2 (#297 built by jenkins on 20140606-2221 git: 80907df78ba6b50c21a598f0caff8b00685d5961)

com.google.common.util.concurrent.UncheckedExecutionException:java.lang.NullPointerException
at LocalCache.java line 2263
in com.google.common.cache.LocalCache$Segment get()

Stack Trace:

1. LocalCache.java line 2263
   in com.google.common.cache.LocalCache$Segment get()
2. LocalCache.java line 4000
   in com.google.common.cache.LocalCache get()
3. LocalCache.java line 4789
   in com.google.common.cache.LocalCache$LocalManualCache get()
4. LandingPageController.java line 1030
   in com.cloudera.server.web.cmf.LandingPageController getHealthInfo()
5. LandingPageController.java line 241
   in com.cloudera.server.web.cmf.LandingPageController servicesTable()
6. line -1
   in sun.reflect.GeneratedMethodAccessor704 invoke()
7. DelegatingMethodAccessorImpl.java line 43
   in sun.reflect.DelegatingMethodAccessorImpl invoke()
8. Method.java line 606
   in java.lang.reflect.Method invoke()
9. HandlerMethodInvoker.java line 176
   in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
10. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
11. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
12. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
13. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
14. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
15. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
16. HttpServlet.java line 595
    in javax.servlet.http.HttpServlet service()
17. HttpServlet.java line 668
    in javax.servlet.http.HttpServlet service()
18. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
19. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
20. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
21. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
22. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
23. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
24. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
25. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
26. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
27. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
28. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
29. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
30. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
31. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
32. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
33. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
34. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
35. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
36. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
37. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
38. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
39. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
40. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
41. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
42. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
43. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
44. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
45. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
46. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
47. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
48. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
50. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
51. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
52. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
53. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
54. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
55. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
56. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
57. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
58. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
59. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
60. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
61. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
62. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
63. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
64. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
65. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
66. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
67. Server.java line 326
    in org.mortbay.jetty.Server handle()
68. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
69. HttpConnection.java line 928
    in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
70. HttpParser.java line 549
    in org.mortbay.jetty.HttpParser parseNext()
71. HttpParser.java line 212
    in org.mortbay.jetty.HttpParser parseAvailable()
72. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
73. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
74. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: java.lang.NullPointerException:
at Preconditions.java line 191
in com.google.common.base.Preconditions checkNotNull()

Stack Trace:

1. Preconditions.java line 191
   in com.google.common.base.Preconditions checkNotNull()
2. AbstractHealthNode.java line 23
   in com.cloudera.cmon.kaiser.graph.AbstractHealthNode <init>()
3. HostHealthNode.java line 27
   in com.cloudera.cmon.kaiser.graph.HostHealthNode <init>()
4. HealthGraph.java line 134
   in com.cloudera.cmon.kaiser.graph.HealthGraph create()
5. LandingPageController.java line 568
   in com.cloudera.server.web.cmf.LandingPageController$HealthInfo <init>()
6. LandingPageController.java line 1035
   in com.cloudera.server.web.cmf.LandingPageController$2 call()
7. LandingPageController.java line 1032
   in com.cloudera.server.web.cmf.LandingPageController$2 call()
8. LocalCache.java line 4792
   in com.google.common.cache.LocalCache$LocalManualCache$1 load()
9. LocalCache.java line 3599
   in com.google.common.cache.LocalCache$LoadingValueReference loadFuture()
10. LocalCache.java line 2379
    in com.google.common.cache.LocalCache$Segment loadSync()
11. LocalCache.java line 2342
    in com.google.common.cache.LocalCache$Segment lockedGetOrLoad()
12. LocalCache.java line 2257
    in com.google.common.cache.LocalCache$Segment get()
13. LocalCache.java line 4000
    in com.google.common.cache.LocalCache get()
14. LocalCache.java line 4789
    in com.google.common.cache.LocalCache$LocalManualCache get()
15. LandingPageController.java line 1030
    in com.cloudera.server.web.cmf.LandingPageController getHealthInfo()
16. LandingPageController.java line 241
    in com.cloudera.server.web.cmf.LandingPageController servicesTable()
17. line -1
    in sun.reflect.GeneratedMethodAccessor704 invoke()
18. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
19. Method.java line 606
    in java.lang.reflect.Method invoke()
20. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
21. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
22. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
23. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
24. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
25. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
26. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
27. HttpServlet.java line 595
    in javax.servlet.http.HttpServlet service()
28. HttpServlet.java line 668
    in javax.servlet.http.HttpServlet service()
29. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
30. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
31. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
32. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
33. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
34. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
35. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
36. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
37. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
38. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
39. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
40. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
41. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
42. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
43. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
44. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
45. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
46. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
47. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
48. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
49. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
50. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
51. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
52. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
53. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
54. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
55. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
56. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
57. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
58. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
59. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
60. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
61. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
62. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
63. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
64. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
65. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
66. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
67. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
68. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
69. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
70. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
71. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
72. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
73. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
74. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
75. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
76. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
77. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
78. Server.java line 326
    in org.mortbay.jetty.Server handle()
79. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
80. HttpConnection.java line 928
    in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
81. HttpParser.java line 549
    in org.mortbay.jetty.HttpParser parseNext()
82. HttpParser.java line 212
    in org.mortbay.jetty.HttpParser parseAvailable()
83. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
84. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
85. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Please help . 

Thanks

Bala

[#2]

Below details present in server.log files .

2014-08-01 03:58:29,707 WARN [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0:resourcepool.BasicResourcePool@1841] com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1cfc6e18 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:
org.postgresql.util.PSQLException: FATAL: the database system is shutting down
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

[#3]

After my datanode stopped abruptly this issue is been coming . Now when i try to add new hosts to cluster , Under Currently managed hosts I see "null" . I am unable to delete this . What should i do to resolve this error 😞

[#4 · Accepted solution]

Problem resolved . It was due to the datanode's network which was abruptly stopped . After revoking the network , the error was gone .