Created 08-27-2021 11:34 AM
Hi Community, I need you support.
I have 3 node-managers and after rebooting 1 node-manager does not start.
(RedHat 7.6 / Hadoop 2.6.5 / Ambari 2.5.1.0).
Added the full log:
https://drive.google.com/file/d/1Osz9nV5PSBAeu8eiiYsJOsib0h3E3Gjk/view?usp=sharing
----------------------------------------------------------
FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: NMWebapps failed to start.
at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:116)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:302)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:547)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)
Caused by: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:402)
at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:100)
... 6 more
Caused by: java.net.BindException: Port in use: 0.0.0.0:8042
at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:983)
at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1006)
at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1063)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:920)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:398)
... 7 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:971)
at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1002)
... 10 more
2021-08-27 14:03:48,701 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at tcolp062.localdomain/10.161.174.132
************************************************************/
Community Thanks!!!!!
Created 08-27-2021 11:47 AM
The log clearly shows that the Address is already in use
Caused by: java.net.BindException: Port in use: 0.0.0.0:8042
Caused by: java.net.BindException: Address already in use
Can you proceed by locating the pid
# lsof -i -P -n | grep LISTEN | grep 8042
Example
# lsof -i:8042
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 9322 yarn 475u IPv4 294790 0t0 TCP *:fs-agent (LISTEN
Kill using the PID
$ kill -9 9322
Restart the service
Please revert
Created 08-27-2021 11:47 AM
The log clearly shows that the Address is already in use
Caused by: java.net.BindException: Port in use: 0.0.0.0:8042
Caused by: java.net.BindException: Address already in use
Can you proceed by locating the pid
# lsof -i -P -n | grep LISTEN | grep 8042
Example
# lsof -i:8042
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 9322 yarn 475u IPv4 294790 0t0 TCP *:fs-agent (LISTEN
Kill using the PID
$ kill -9 9322
Restart the service
Please revert
Created 08-27-2021 12:11 PM
Thanks you very much @Shelton. You are the best!