Docker HDP Sandbox fails to start with " ... port already allocated" error

[#1]

$ sh start_sandbox-hdp.sh
Waiting for docker daemon to start up:
7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59\
docker: Error response from daemon: driver failed programming external connectivity on endpoint sandbox-hdp (a2149c348b6f3e4fc8d916baa20162fe6f6a01e6e4c40b2de64624b8ea1cf8b8): Error starting userland proxy: Bind for 0.0.0.0:8443 failed: port is already allocated.
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Waiting for ambari agent to connect
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Waiting for ambari services to start 
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running
Error response from daemon: Container 7a49c78991d64342d07657ceaff1e571e1836f0d041479def601540d2311cd59 is not running

Started Hortonworks HDP container

Found the issue to be:

$ sudo lsof -nP | grep LISTEN
...
AWRemoteM  1250             root    4u     IPv4 0x30c3471b54f7930b         0t0      TCP *:8443 (LISTEN)

I cannot stop the application that's using the port so need another solution.

I then modified the startup script by changing the port mapping from 8443:8443 to 8444:8444 and it successfully started with some other issues like welcome page (localhost:8888) and some others not opening up (may or may not be related). Ambari was accessible though (localhost:8080).

Then changed the port mapping to 8444:8443 and started it but startup hung on:

...
Ambari Agent successfully started
Agent PID at: /var/run/ambari-agent/ambari-agent.pid
Agent out at: /var/log/ambari-agent/ambari-agent.out
Agent log at: /var/log/ambari-agent/ambari-agent.log
Starting shellinaboxd:                                     [  OK  ]
Waiting for ambari agent to connect
.................................................

I see similar issues being reported but not together so looking for solutions.

I have encountered this problem with multiple versions of the Sandbox and Docker (going back at least a year).

[#2]

I am not sure on the versions, but there was an issue with docker not freeing ports that were in use by an already stopped container. So you can try to restart docker, change the port mapping 8444:8443 and see if it starts up.

[#3]

Thanks for your reply. From what I've researched that issue was already resolved in earlier versions so that should not be an issue. I'll try a clean start to make sure anyway.

I did try 8444:8443 and the port error went away but the startup hangs as I've reported above. I'll give it a go again and see if I move any further.

[#4]

@Anjal Naik

As you mentioned that you cannot stop the application that's using the port 8443 so in that case you will have to add some additional ports to the sandbox docker.

You can refer to the following article to know about how to add some additional ports to Docker. and then map your altered ports to it.

https://community.hortonworks.com/articles/65914/how-to-add-ports-to-the-hdp-25-virtualbox-sandbox.h...

[#5]

Thanks for your reply. Yes, I thought the same too and have tried a couple of solutions as I explained but failing at the stages.

[#6]

@Anjal Naik

Try looking at the ambari agent logs in /var/log/ambari-agent/ to see why it has a hard time connecting. Perhaps there is something there to explain the behavior.

[#7]

The simplest solution to really isolate sandbox is to use Vagrant/Virtualbox image. I recently tried to setup docker HDP and encountered same problem as you described, but with port 2049.

nfsd 2924 root 9u IPv4 0x467a33f127881a43 0t0 TCP *:2049 (LISTEN)
nfsd 2924 root 10u IPv6 0x467a33f11dc28473 0t0 TCP *:2049 (LISTEN)

I am on Mac Yosemite.

Previously I had been using VirtualBox version of HDP. The only problem that I had was long loading time (happened because VM lacked memory), but after I fine-tuned VM, everything is working fast and stable. Easiest way.