CDH5.5.1 failed to start history server

[#1]

Hi:
I used to have CDH5.4.7.  I could start all hadoop yarn servers O.K. in pseudo distributed mode.  I just upgraded to 5.5 today.  I found that I couldn't start history server now.  The error is like the followings.   I checked 'netstat'.  Port 19888 is not binded to any process.  If I changed mapreduce.jobhistory.webapp.address in mapred-site.xml to 19088.   I would get the similar error with Port in use: 19088.  Please advise how I can resolve this issue or is this a bug.Thanks.

6/01/02 15:15:52 INFO service.AbstractService: Service HistoryClientService failed in state STARTED; cause: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:278)
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.initializeWebApp(HistoryClientService.java:156)
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.serviceStart(HistoryClientService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceStart(JobHistoryServer.java:197)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:223)
at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:232)
Caused by: java.net.BindException: Port in use: ununtu:19888
at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:944)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:880)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:273)
... 8 more
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:939)
... 10 more

[#2]

Just a wild guess: is IPv6 disabled?

• -Djava.net.preferIPv4Stack=true in the HS process arguments
• or no IPv6 addresses on the interfaces

You could try to set this just to be sure (but I have this set to "0" and it is working fine anyway):

•  sysctl net.ipv6.bindv6only = 1

[#3 · Accepted solution]

I didn't make any changes. Somehow it is working now. 

[#4]

I had the same issue. The solution was to start the history server first. This means before the other cdh services like hdfs and yarn are started.