Failed binding http info server to port: 60030

[#1]

Hi,

In Cloudera Manager one of our HBase region servers won't start.  We get the following message in the logs but can't figure out why it's just this region server and no others and what it is that's blocking it.

Has anyone seen this before?  It was working ok but over the weekend seems to have stopped.

org.apache.hadoop.hbase.regionserver.HRegionServer Failed binding http info server to port: 60030

Caused by: java.net.BindException: Address already in use

Thanks

Mark

[#2]

Error

Region server exiting

java.lang.RuntimeException: Failed construction of Regionserver: class org.apache.hadoop.hbase.regionserver.HRegionServer

         at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2452)

         at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:61)

         at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)

         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

         at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)

         at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2467)

Caused by: java.lang.reflect.InvocationTargetException

         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

         at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2450)

         ... 5 more

Caused by: java.net.BindException: Port in use: 0.0.0.0:60030

         at org.apache.hadoop.http.HttpServer.openListener(HttpServer.java:783)

         at org.apache.hadoop.http.HttpServer.start(HttpServer.java:727)

         at org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1725)

         at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:638)

         ... 10 more

Caused by: java.net.BindException: Address already in use

         at sun.nio.ch.Net.bind0(Native Method)

         at sun.nio.ch.Net.bind(Net.java:444)

         at sun.nio.ch.Net.bind(Net.java:436)

         at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)

         at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)

         at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)

         at org.apache.hadoop.http.HttpServer.openListener(HttpServer.java:779)

         ... 13 more

[#3]

Port numbers larger than 32765 (ephemeral ports) can be in use at any
time. It is possible another process is holding the port.

This should tell you the PID that holds the port. That should help
decide next steps
# netstat -nap | grep 60030

[#4 · Accepted solution]

Thanks for the reply GautamG

My colleague has a resolved this by restarting all the services on the machine.  I had tried just restarting HBase but that hadn't worked.

Cheers

Mark