Unable to log in to Ambari UI - NPE\500 Server Error

[#1]

I have installed ambari-server 2.2.0 on CentOS 7 using all default responses to ambari-server setup. I have not added new users, nor have I changed the password of 'admin' from the default. I was able to login to the UI initially, and use the REST API to submit blueprints and cluster. The cluster was created correctly and all installed hadoop services are running fine. The stack is HDP 2.3. However, after a few days, I am no longer able to log in to the Ambari UI. When I land on the login page and try to login as 'admin', I see the following message:

Unable to connect to Ambari Server. Confirm Ambari Server is running and you can reach Ambari Server from this machine.

When I check /var/log/ambari-server/ambari-server.log I see the following stack trace:

29 Feb 2016 11:18:17,234  WARN [qtp-ambari-client-23] ServletHandler:563 - /api/v1/users/admin
java.lang.NullPointerException
        at org.apache.ambari.server.controller.internal.ActiveWidgetLayoutResourceProvider.getResources(ActiveWidgetLayoutResourceProvider.java:153)
        at org.apache.ambari.server.controller.internal.ClusterControllerImpl$ExtendedResourceProviderWrapper.queryForResources(ClusterControllerImpl.java:945)
        at org.apache.ambari.server.controller.internal.ClusterControllerImpl.getResources(ClusterControllerImpl.java:132)
        at org.apache.ambari.server.api.query.QueryImpl.doQuery(QueryImpl.java:508)
        at org.apache.ambari.server.api.query.QueryImpl.queryForSubResources(QueryImpl.java:463)
        at org.apache.ambari.server.api.query.QueryImpl.queryForResources(QueryImpl.java:436)
        at org.apache.ambari.server.api.query.QueryImpl.execute(QueryImpl.java:216)
        at org.apache.ambari.server.api.handlers.ReadHandler.handleRequest(ReadHandler.java:68)
        at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:106)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:75)
        at org.apache.ambari.server.api.services.UserService.getUser(UserService.java:68)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:196)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
        at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
        at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
        at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:216)
        at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:205)
        at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745) 

I really need to know how to recover from this. The hadoop services are all running fine for now, but I need the Ambari UI for management\alerts, etc. I would like to not have to recreate cluster as it is already in use. **EDIT** I should add that I am still able to make REST API requests directly using an app like POSTMAN, or curl. for instance: /api/v1/clusters/{name} returns the correct information when using admin\admin as the auth.

[#2]

@Heath Abelson

Login to Ambari server and run

ambari-server start --> I believe server is up as you can make rest calls

I see that you cannot reset the ambari database as cluster is in use already. Do you have access to support?

[#3]

@Neeraj Sabharwal

yes, the service is running, and restarting the service does not seem to help. As you point out, I am still able to access the the rest calls.

I tried to poke around the postgres database to see if the user table was maybe corrupted, but it looks ok

I don't think I have access to support. If that is a paid service, then I don't.

[#4]

@Heath Abelson

There are a couple of check to do know they could be painful processes to go through again

To troubleshoot check that the Ambari on system reboot is up and running

$ambari-server status 
$ambari-agent status 

Its advisable to check the backend database status just to make sure service postgresql status

Stop gracefully the Ambari Server

$ambari-server stop 
$ambari-agent stop 

On the Ambari Server host machine:

$ambari-server start 

Restart the agent on each host in your cluster:

$ambari-agent start 

To check if the service has started and is running without any issues check the Ambari Server logs:

$ /var/log/ambari-server/ambari-server.log 

Ambari Agent logs on any host with an Ambari Agent:

$ /var/log/ambari-agent/ambari-agent.log 

Then check the availability of the Ambari ports

$netstat -anp | grep 8080 
$lsof -i | grep 8080

[#5]

I have performed all those actions. The server starts and is running with no errors. Ports look ok (again, I am able to still interact with the REST API). Postgres also seems fine ans I am able to login and check the user table.

I haven't spent much time checking the status of the agents, as I didn't think their status could in any way interfere with logging into the server.

The only error I see in the server log is the one in my original post, which occurs on login attempt.

[#6]

@Heath Abelson

please take a look at this solution, does it help you at all? https://community.hortonworks.com/questions/20603/500-status-code-received-on-get-method-for-api-api...

[#7]

The link is appreciated, but it doesn't appear to be related. My ambari-server is running as root.

[#8]

@Heath Abelson before you login to Ambari UI, try hard reset of the browser cache by hitting CTRL-SHIFT-R. After that try to login again.

[#9]

@Artem Ervits no dice. Even if I try to login from a browser on different machines, same result.

Is it possible that backing up the DB and reinstalling the server might help? I was hoping to not have to get so drastic

[#10]

chance is you will restore the backup and this inconsistency will crop up. Tagging some Ambari experts @smohanty @yusaku @mahadev