Restarting ambari server causing "Address already in use" in AWS Cloudbreak

[#1]

Problem

When running

$ ambari-server restart

I can see "tones" of these messages in ambari-server.log

27 Feb 2017 11:38:00,942  WARN [main] AbstractLifeCycle:204 - FAILED [email protected]:8080: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:463)
        at sun.nio.ch.Net.bind(Net.java:455)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.eclipse.jetty.server.Server.doStart(Server.java:293)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:616)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:925)
27 Feb 2017 11:38:00,942  WARN [main] AbstractLifeCycle:204 - FAILED [email protected]:8080: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:463)
        at sun.nio.ch.Net.bind(Net.java:455)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.eclipse.jetty.server.Server.doStart(Server.java:293)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:616)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:925)
27 Feb 2017 11:38:00,944  WARN [main] AbstractLifeCycle:204 - FAILED org.eclipse.jetty.server.Server@5c0ecfc4: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:463)
        at sun.nio.ch.Net.bind(Net.java:455)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)

Solution

On AWS Cloudbreak starts the instances with Amazon Linux. On Amazon-linux Upstart is used for system init. To restart ambari server with upstart you have to use "initctl" command like:

$ initctl restart ambari-server

If you have ambari server already running, do the following after logging in to ambari server node as root:

a) $ ambari-server stop
b) $ ps -ef | grep ambari-server
c) kill all the outstanding ambari-server's processes
d) backup and remove all logs from /var/log/ambari-server 
e) $ initctl start ambari-server 
f) tail -f /var/log/ambari-server/ambari-server.log

Now, there is no more "Address already in use" errors in ambari-server.log.