Support Questions

Find answers, ask questions, and share your expertise

Ambari Server Restarted, now Web Interface Won't Load

Explorer

I have a cluster that has been running pretty smoothly for over a month now that is rather large. I had just fixed an issue with a few DataNodes and was left with the web interface saying that 5 hosts were in maintenance mode when they were not. I tried turning mm on/off but it didn't help so I finally tried a sudo ambari-server restart

Afterwards, I tried to login again and the web interface wouldn't load. Waited a few minutes, still nada.

Looking at today's entries in /var/log/ambari-server/ambari-server.log had errors regarding the metric server but nothing that seemed relevant. The network settings are all the same and haven't changed, I am not sure what went wrong? Any ideas of what logs I should read or what I should try?

EDIT:

Thanks for the fast replies!!

I have attached the info requested as files so this post doesn't extend down a mile.

It looks like the ambari service is not listening on port 8080.

EDIT2:

I ended up rolling back the ambari server vm to an earlier checkpoint and luckily there weren't too many errors to deal with. I will update again if the issue comes back.

1 ACCEPTED SOLUTION

Could you also restart the ambari agents of your cluster. It'd be good to have some of the errors that you saw in the ambari-server log, could you post some of the log or upload the log file?

After you restarted the ambari server, did it show up in the process (ps -aux | grep ambari-server), also make sure you see the ambari process listening on port 8080 (netstat -anop)

View solution in original post

34 REPLIES 34

Could you also restart the ambari agents of your cluster. It'd be good to have some of the errors that you saw in the ambari-server log, could you post some of the log or upload the log file?

After you restarted the ambari server, did it show up in the process (ps -aux | grep ambari-server), also make sure you see the ambari process listening on port 8080 (netstat -anop)

Explorer

Thanks for the quick reply, it looks like the ambari process isnt listening on 8080.

Mentor

@Kyle Pifer is the port in use? Ambari is not running?

Explorer

ambari-server status shows that it is running and the port 8080 does not appear to be in use.

Since ambari is not listening on port 8080, there is something with the ambari server startup. Could be that the database service is down or the ambari server port is blocked.

Open the ambari-server log (tail -fn 100 /var/log/ambari-server/ambari-server.log) and use a second console to start the ambari-server (ambari-server start). Check the log in the other window and see if there are any errors 🙂

Explorer

There weren't any errors that I saw. Just warning messages about one of the nodes losing it's heartbeat.

tail -fn 100 /var/log/ambari-server/ambari-server.log >> ambari-server-log-restart.txt

Thanks @Kyle Pifer, this is strange. Looking at your ps aux output from above, ambari server is running and even established a connection to the database, but I dont see it listening on any port for incoming connections.

Did you enable wire encryption (ssl) for ambari?

Explorer

When I deployed ambari/hadoop initially, I left most all of the settings as default with the intent to adjust them later as our team would like if we continue to use the product. I don't remember explicitly enabling it, so I imagine ssl is not enabled.

Mentor

Try a different browser or ctrl-shft-r on the original to clear cache. @Kyle Pifer

Explorer

I had tried that initially, but no luck

@Kyle Pifer Please attach /etc/ambari-server/conf/ambari.properties

Explorer

@Kyle Pifer Thanks for sharing the ambari.properties. See this article

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_reference_guide/content/_optional...

You are using default port for sure otherwise you should have client.api.port defined.

Steps:

ambari-server stop

mv/var/log/ambari-server/ambari-server.log /var/log/ambari-server/ambari-server.log.bakup

ambari-server start

now, check whats in the log file.

Explorer

The log file looks the same. I tried defining a different port which didnt work, then I left it defined in ambari.properties as 8080 as the link you provided said but it still isnt working. Is there a problem with using port 8080?

@Kyle Pifer This is very strange. No there is no problem on using problem 8080.

Whats in the log when hit the browser? netstat -anp | grep 8080

Explorer

Thats the problem, netstat shows nothing for p 8080.

@Kyle Pifer I just did the same exercise in my cluster and I see this http://pastebin.com/008iBDuw

Wait for sometime and see whats coming in ...

restart agents , ambari and run tail -f /var/log/ambari-server/ambari-server.log

@Kyle Pifer Whats the output of ambari-server start ?

Explorer

Still nothing in ambari-server log, but the agent logs are showing connection refused to ambariserver:8440

@Kyle Pifer Do you have access to support?

Is it production env?

Ambari ServerAmbari Server host8440httpsHandshake Port for Ambari Agents to Ambari Server