Support Questions
Find answers, ask questions, and share your expertise

HDP Services Port Collision

New Contributor

When restarting our HDP cluster today, one of the NodeManager for Yarn was unable to start. After sleuthing to the node it turns out that it was a simple port collision that occurred. It turns out that the Ambari Metrics process HMaster process had grabbed port 45454 and this blocked the NodeManager from starting.

I'm not exactly sure how to reproduce this as we have restarted our cluster dozen's of times without incidence but it might be something straight forward to retry the NodeManager on a different port if it is in use.

Here are the steps I use to fix the problem:

  1. Determine what the issue was with the NodeManager. Here is the stacktrace:
    Problem binding to [] Address already in use; For more details see:
    org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Problem binding to [] Address already in use; For more details see:
      at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(
      at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(
      at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(
      at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(
      at org.apache.hadoop.service.AbstractService.start(
      at org.apache.hadoop.service.CompositeService.serviceStart(
      at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(
      at org.apache.hadoop.service.AbstractService.start(
      at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(
      at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(
  2. I then determined the process that was using the port: org.apache.hadoop.hbase.master.HMaster
  3. I then within Ambari stopped the the Ambari Metrics service.
  4. I then successfully started the NodeManager.
  5. I restarted the Ambari Metrics service.
  6. Happy cluster!

Here are the versions of the offending software we are using:

Stack HDP-2.5
Name: HDP-
YARN: 2.7.3
Ambari Metrics: 0.1.0