Ambari Server Restarted, now Web Interface Won't Load

[#1]

I have a cluster that has been running pretty smoothly for over a month now that is rather large. I had just fixed an issue with a few DataNodes and was left with the web interface saying that 5 hosts were in maintenance mode when they were not. I tried turning mm on/off but it didn't help so I finally tried a sudo ambari-server restart

Afterwards, I tried to login again and the web interface wouldn't load. Waited a few minutes, still nada.

Looking at today's entries in /var/log/ambari-server/ambari-server.log had errors regarding the metric server but nothing that seemed relevant. The network settings are all the same and haven't changed, I am not sure what went wrong? Any ideas of what logs I should read or what I should try?

EDIT:

Thanks for the fast replies!!

I have attached the info requested as files so this post doesn't extend down a mile.

It looks like the ambari service is not listening on port 8080.

EDIT2:

I ended up rolling back the ambari server vm to an earlier checkpoint and luckily there weren't too many errors to deal with. I will update again if the issue comes back.

[#2 · Accepted solution]

Could you also restart the ambari agents of your cluster. It'd be good to have some of the errors that you saw in the ambari-server log, could you post some of the log or upload the log file?

After you restarted the ambari server, did it show up in the process (ps -aux | grep ambari-server), also make sure you see the ambari process listening on port 8080 (netstat -anop)

[#3]

Thanks for the quick reply, it looks like the ambari process isnt listening on 8080.

[#4]

@Kyle Pifer is the port in use? Ambari is not running?

[#5]

ambari-server status shows that it is running and the port 8080 does not appear to be in use.

[#6]

Since ambari is not listening on port 8080, there is something with the ambari server startup. Could be that the database service is down or the ambari server port is blocked.

Open the ambari-server log (tail -fn 100 /var/log/ambari-server/ambari-server.log) and use a second console to start the ambari-server (ambari-server start). Check the log in the other window and see if there are any errors 🙂

[#7]

There weren't any errors that I saw. Just warning messages about one of the nodes losing it's heartbeat.

tail -fn 100 /var/log/ambari-server/ambari-server.log >> ambari-server-log-restart.txt

[#8]

Thanks @Kyle Pifer, this is strange. Looking at your ps aux output from above, ambari server is running and even established a connection to the database, but I dont see it listening on any port for incoming connections.

Did you enable wire encryption (ssl) for ambari?

[#9]

When I deployed ambari/hadoop initially, I left most all of the settings as default with the intent to adjust them later as our team would like if we continue to use the product. I don't remember explicitly enabling it, so I imagine ssl is not enabled.

[#10]

Try a different browser or ctrl-shft-r on the original to clear cache. @Kyle Pifer