Ambari Server Restarted, now Web Interface Won't Load

[#1]

I have a cluster that has been running pretty smoothly for over a month now that is rather large. I had just fixed an issue with a few DataNodes and was left with the web interface saying that 5 hosts were in maintenance mode when they were not. I tried turning mm on/off but it didn't help so I finally tried a sudo ambari-server restart

Afterwards, I tried to login again and the web interface wouldn't load. Waited a few minutes, still nada.

Looking at today's entries in /var/log/ambari-server/ambari-server.log had errors regarding the metric server but nothing that seemed relevant. The network settings are all the same and haven't changed, I am not sure what went wrong? Any ideas of what logs I should read or what I should try?

EDIT:

Thanks for the fast replies!!

I have attached the info requested as files so this post doesn't extend down a mile.

It looks like the ambari service is not listening on port 8080.

EDIT2:

I ended up rolling back the ambari server vm to an earlier checkpoint and luckily there weren't too many errors to deal with. I will update again if the issue comes back.

[#2]

I had tried that initially, but no luck

[#3]

@Kyle Pifer Please attach /etc/ambari-server/conf/ambari.properties

[#4]

ambariproperties.txt

[#5]

@Kyle Pifer Thanks for sharing the ambari.properties. See this article

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_reference_guide/content/_optional...

You are using default port for sure otherwise you should have client.api.port defined.

Steps:

ambari-server stop

mv/var/log/ambari-server/ambari-server.log /var/log/ambari-server/ambari-server.log.bakup

ambari-server start

now, check whats in the log file.

[#6]

The log file looks the same. I tried defining a different port which didnt work, then I left it defined in ambari.properties as 8080 as the link you provided said but it still isnt working. Is there a problem with using port 8080?

[#7]

@Kyle Pifer This is very strange. No there is no problem on using problem 8080.

Whats in the log when hit the browser? netstat -anp | grep 8080

[#8]

Thats the problem, netstat shows nothing for p 8080.

[#9]

@Kyle Pifer I just did the same exercise in my cluster and I see this http://pastebin.com/008iBDuw

Wait for sometime and see whats coming in ...

restart agents , ambari and run tail -f /var/log/ambari-server/ambari-server.log

[#10]

@Kyle Pifer Whats the output of ambari-server start ?

[#11]

Still nothing in ambari-server log, but the agent logs are showing connection refused to ambariserver:8440