After restart server: nodemanager not start

[#1]

Hi Community, I need you support.

I have 3 node-managers and after rebooting 1 node-manager does not start.

(RedHat 7.6 / Hadoop 2.6.5 / Ambari 2.5.1.0).

Added the full log:

https://drive.google.com/file/d/1Osz9nV5PSBAeu8eiiYsJOsib0h3E3Gjk/view?usp=sharing

----------------------------------------------------------

FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: NMWebapps failed to start.
at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:116)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:302)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:547)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)
Caused by: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:402)
at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:100)
... 6 more
Caused by: java.net.BindException: Port in use: 0.0.0.0:8042
at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:983)
at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1006)
at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1063)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:920)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:398)
... 7 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:971)
at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1002)
... 10 more
2021-08-27 14:03:48,701 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at tcolp062.localdomain/10.161.174.132
************************************************************/

Community Thanks!!!!!

[#2 · Accepted solution]

@vciampa 

The log clearly shows that the Address is already in use

Caused by: java.net.BindException: Port in use: 0.0.0.0:8042
Caused by: java.net.BindException: Address already in use

Can you proceed by locating the pid

# lsof -i -P -n | grep LISTEN | grep 8042

Example

# lsof -i:8042

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 9322 yarn 475u IPv4 294790 0t0 TCP *:fs-agent (LISTEN

Kill using the PID

$ kill -9 9322

Restart the service

Please revert

[#3]

Thanks you very much @Shelton. You are the best!