question Re: Ambari Server Restarted, now Web Interface Won't Load in Support Questions

Ambari Server Restarted, now Web Interface Won't Load

v-kypife — Sat, 30 Jan 2016 15:48:49 GMT

I have a cluster that has been running pretty smoothly for over a month now that is rather large. I had just fixed an issue with a few DataNodes and was left with the web interface saying that 5 hosts were in maintenance mode when they were not. I tried turning mm on/off but it didn't help so I finally tried a sudo ambari-server restart

Afterwards, I tried to login again and the web interface wouldn't load. Waited a few minutes, still nada.

Looking at today's entries in /var/log/ambari-server/ambari-server.log had errors regarding the metric server but nothing that seemed relevant. The network settings are all the same and haven't changed, I am not sure what went wrong? Any ideas of what logs I should read or what I should try?

EDIT:

Thanks for the fast replies!!

I have attached the info requested as files so this post doesn't extend down a mile.

It looks like the ambari service is not listening on port 8080.

EDIT2:

I ended up rolling back the ambari server vm to an earlier checkpoint and luckily there weren't too many errors to deal with. I will update again if the issue comes back.

Re: Ambari Server Restarted, now Web Interface Won't Load

jstraub — Sat, 30 Jan 2016 16:37:55 GMT

Could you also restart the ambari agents of your cluster. It'd be good to have some of the errors that you saw in the ambari-server log, could you post some of the log or upload the log file?

After you restarted the ambari server, did it show up in the process (ps -aux | grep ambari-server), also make sure you see the ambari process listening on port 8080 (netstat -anop)

Re: Ambari Server Restarted, now Web Interface Won't Load

aervits — Sat, 30 Jan 2016 20:15:01 GMT

Try a different browser or ctrl-shft-r on the original to clear cache. @Kyle Pifer

Re: Ambari Server Restarted, now Web Interface Won't Load

v-kypife — Sun, 31 Jan 2016 03:34:34 GMT

Thanks for the quick reply, it looks like the ambari process isnt listening on 8080.

Re: Ambari Server Restarted, now Web Interface Won't Load

aervits — Sun, 31 Jan 2016 03:50:57 GMT

@Kyle Pifer is the port in use? Ambari is not running?

Re: Ambari Server Restarted, now Web Interface Won't Load

v-kypife — Sun, 31 Jan 2016 04:06:14 GMT

ambari-server status shows that it is running and the port 8080 does not appear to be in use.

Re: Ambari Server Restarted, now Web Interface Won't Load

v-kypife — Sun, 31 Jan 2016 04:07:03 GMT

I had tried that initially, but no luck

Re: Ambari Server Restarted, now Web Interface Won't Load

jstraub — Sun, 31 Jan 2016 04:09:01 GMT

Since ambari is not listening on port 8080, there is something with the ambari server startup. Could be that the database service is down or the ambari server port is blocked.

Open the ambari-server log (tail -fn 100 /var/log/ambari-server/ambari-server.log) and use a second console to start the ambari-server (ambari-server start). Check the log in the other window and see if there are any errors 🙂

Re: Ambari Server Restarted, now Web Interface Won't Load

v-kypife — Sun, 31 Jan 2016 04:47:34 GMT

There weren't any errors that I saw. Just warning messages about one of the nodes losing it's heartbeat.

tail -fn 100 /var/log/ambari-server/ambari-server.log >> ambari-server-log-restart.txt

Re: Ambari Server Restarted, now Web Interface Won't Load

nsabharwal — Sun, 31 Jan 2016 04:57:41 GMT

@Kyle Pifer Please attach /etc/ambari-server/conf/ambari.properties

Re: Ambari Server Restarted, now Web Interface Won't Load

v-kypife — Sun, 31 Jan 2016 05:01:11 GMT

ambariproperties.txt

Re: Ambari Server Restarted, now Web Interface Won't Load

nsabharwal — Sun, 31 Jan 2016 05:09:52 GMT

@Kyle Pifer Thanks for sharing the ambari.properties. See this article

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_reference_guide/content/_optional_changing_the_default_ambari_server_port.html

You are using default port for sure otherwise you should have client.api.port defined.

Steps:

ambari-server stop

mv/var/log/ambari-server/ambari-server.log /var/log/ambari-server/ambari-server.log.bakup

ambari-server start

now, check whats in the log file.

Re: Ambari Server Restarted, now Web Interface Won't Load

jstraub — Sun, 31 Jan 2016 05:43:01 GMT

Thanks @Kyle Pifer, this is strange. Looking at your ps aux output from above, ambari server is running and even established a connection to the database, but I dont see it listening on any port for incoming connections.

Did you enable wire encryption (ssl) for ambari?

Re: Ambari Server Restarted, now Web Interface Won't Load

v-kypife — Sun, 31 Jan 2016 05:50:09 GMT

The log file looks the same. I tried defining a different port which didnt work, then I left it defined in ambari.properties as 8080 as the link you provided said but it still isnt working. Is there a problem with using port 8080?

Re: Ambari Server Restarted, now Web Interface Won't Load

v-kypife — Sun, 31 Jan 2016 05:52:42 GMT

When I deployed ambari/hadoop initially, I left most all of the settings as default with the intent to adjust them later as our team would like if we continue to use the product. I don't remember explicitly enabling it, so I imagine ssl is not enabled.

Re: Ambari Server Restarted, now Web Interface Won't Load

nsabharwal — Sun, 31 Jan 2016 05:53:22 GMT

@Kyle Pifer This is very strange. No there is no problem on using problem 8080.

Whats in the log when hit the browser? netstat -anp | grep 8080

Re: Ambari Server Restarted, now Web Interface Won't Load

v-kypife — Sun, 31 Jan 2016 05:54:45 GMT

Thats the problem, netstat shows nothing for p 8080.

Re: Ambari Server Restarted, now Web Interface Won't Load

nsabharwal — Sun, 31 Jan 2016 05:58:04 GMT

@Kyle Pifer I just did the same exercise in my cluster and I see this http://pastebin.com/008iBDuw

Wait for sometime and see whats coming in ...

restart agents , ambari and run tail -f /var/log/ambari-server/ambari-server.log

Re: Ambari Server Restarted, now Web Interface Won't Load

nsabharwal — Sun, 31 Jan 2016 06:02:20 GMT

@Kyle Pifer Whats the output of ambari-server start ?

Re: Ambari Server Restarted, now Web Interface Won't Load

aervits — Sun, 31 Jan 2016 07:40:44 GMT

@Kyle Pifer please start ambari server with debug switch and analyze the output. Link