A server error has occurred. (java.lang.NullPointerException at LocalCache.java line 2263)

[#1]

The below error appears after logging in the cloudera manager.

Following a restart of the cloudera manager service the error remains.

Has anyone got any helpful intel?

A server error has occurred. Send the following information to Cloudera.

Path: http://*****************:****/cmf/services/landingPageStatusContent

Version: Cloudera Enterprise 5.0.1 (#257 built by jenkins on 20140512-1726 git: 7919cb55f527d5c6285bc455c93e9f74ebbc37e3)

com.google.common.util.concurrent.UncheckedExecutionException:java.lang.NullPointerException
at LocalCache.java line 2263
in com.google.common.cache.LocalCache$Segment get()

Stack Trace:

1. LocalCache.java line 2263
   in com.google.common.cache.LocalCache$Segment get()
2. LocalCache.java line 4000
   in com.google.common.cache.LocalCache get()
3. LocalCache.java line 4789
   in com.google.common.cache.LocalCache$LocalManualCache get()
4. LandingPageController.java line 1030
   in com.cloudera.server.web.cmf.LandingPageController getHealthInfo()
5. LandingPageController.java line 241
   in com.cloudera.server.web.cmf.LandingPageController servicesTable()
6. line -1
   in sun.reflect.GeneratedMethodAccessor674 invoke()
7. DelegatingMethodAccessorImpl.java line 43
   in sun.reflect.DelegatingMethodAccessorImpl invoke()
8. Method.java line 606
   in java.lang.reflect.Method invoke()
9. HandlerMethodInvoker.java line 176
   in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
10. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
11. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
12. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
13. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
14. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
15. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
16. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
17. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
18. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
19. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
20. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
21. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
22. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
23. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
24. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
25. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
26. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
27. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
28. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
29. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
30. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
31. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
32. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
33. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
34. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
35. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
36. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
37. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
38. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
39. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
40. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
41. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
42. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
43. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
44. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
45. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
46. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
47. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
48. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
50. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
51. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
52. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
53. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
54. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
55. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
56. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
57. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
58. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
59. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
60. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
61. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
62. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
63. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
64. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
65. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
66. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
67. Server.java line 326
    in org.mortbay.jetty.Server handle()
68. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
69. HttpConnection.java line 928
    in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
70. HttpParser.java line 549
    in org.mortbay.jetty.HttpParser parseNext()
71. HttpParser.java line 212
    in org.mortbay.jetty.HttpParser parseAvailable()
72. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
73. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
74. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: java.lang.NullPointerException:
at Preconditions.java line 191
in com.google.common.base.Preconditions checkNotNull()

Stack Trace:

1. Preconditions.java line 191
   in com.google.common.base.Preconditions checkNotNull()
2. AbstractHealthNode.java line 23
   in com.cloudera.cmon.kaiser.graph.AbstractHealthNode <init>()
3. HostHealthNode.java line 27
   in com.cloudera.cmon.kaiser.graph.HostHealthNode <init>()
4. HealthGraph.java line 134
   in com.cloudera.cmon.kaiser.graph.HealthGraph create()
5. LandingPageController.java line 568
   in com.cloudera.server.web.cmf.LandingPageController$HealthInfo <init>()
6. LandingPageController.java line 1035
   in com.cloudera.server.web.cmf.LandingPageController$2 call()
7. LandingPageController.java line 1032
   in com.cloudera.server.web.cmf.LandingPageController$2 call()
8. LocalCache.java line 4792
   in com.google.common.cache.LocalCache$LocalManualCache$1 load()
9. LocalCache.java line 3599
   in com.google.common.cache.LocalCache$LoadingValueReference loadFuture()
10. LocalCache.java line 2379
    in com.google.common.cache.LocalCache$Segment loadSync()
11. LocalCache.java line 2342
    in com.google.common.cache.LocalCache$Segment lockedGetOrLoad()
12. LocalCache.java line 2257
    in com.google.common.cache.LocalCache$Segment get()
13. LocalCache.java line 4000
    in com.google.common.cache.LocalCache get()
14. LocalCache.java line 4789
    in com.google.common.cache.LocalCache$LocalManualCache get()
15. LandingPageController.java line 1030
    in com.cloudera.server.web.cmf.LandingPageController getHealthInfo()
16. LandingPageController.java line 241
    in com.cloudera.server.web.cmf.LandingPageController servicesTable()
17. line -1
    in sun.reflect.GeneratedMethodAccessor674 invoke()
18. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
19. Method.java line 606
    in java.lang.reflect.Method invoke()
20. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
21. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
22. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
23. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
24. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
25. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
26. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
27. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
28. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
29. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
30. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
31. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
32. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
33. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
34. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
35. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
36. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
37. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
38. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
39. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
40. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
41. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
42. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
43. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
44. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
45. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
46. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
47. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
48. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
49. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
50. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
51. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
52. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
53. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
54. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
55. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
56. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
57. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
58. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
59. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
60. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
61. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
62. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
63. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
64. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
65. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
66. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
67. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
68. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
69. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
70. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
71. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
72. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
73. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
74. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
75. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
76. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
77. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
78. Server.java line 326
    in org.mortbay.jetty.Server handle()
79. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
80. HttpConnection.java line 928
    in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
81. HttpParser.java line 549
    in org.mortbay.jetty.HttpParser parseNext()
82. HttpParser.java line 212
    in org.mortbay.jetty.HttpParser parseAvailable()
83. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
84. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
85. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

[#2 · Accepted solution]

Resolved.

It was a buggy entry in the HOSTS table.

Just execute:

select *
from HOSTS
where ip_address is null or name is null;

Grab the host_id and update the record just in case removing it does any bad juju.

update HOSTS set name = 'ignoreme', ip_address = '0.0.0.0' where host_id = your_host_id_here;

[#3]

Hello!

I'm relatively new to the Hadoop ecosystem. Can you tell me from where I would go about querying against the hosts table you mention? Is it a Zookeeper thing? Hive? Any advice or direction towards documentation is greatly appreciated. Thanks!

[#4]

Found it myself, but in case someone else has the same question I did:

On a default CDH install, check /var/lib/cloudera-scm-server-db/data/generated-password.txt for the DB password. Then:

psql -h 127.0.0.1 -p 7432 -U cloudera-scm -W scm

HTH!

[#5]

I am seeing an empty set of hosts

psql (8.4.20)
Type "help" for help.

scm=# select *
scm-# from HOSTS
scm-# where ip_address is null or name is null;
host_id | optimistic_lock_version | host_identifier | name | ip_address | rack_id | status | config_container_id | mainten
ance_count | decommission_count | cluster_id | num_cores | total_phys_mem_bytes | public_name | public_ip_address | cloud_p
rovider
---------+-------------------------+-----------------+------+------------+---------+--------+---------------------+--------
-----------+--------------------+------------+-----------+----------------------+-------------+-------------------+--------
--------
(0 rows)