Datanode won't start - dfs.datanode.http.address in use

Mahadi — Fri, 13 Aug 2021 00:23:30 GMT

One of of my data-node won't start. The error log on {HADOOP_LOG}/hdfs/{datanode}.err shown that the port of dfs.datanode.http.address already in use. We use custom port for dfs.datanode.http.address. When I run netstat -lpn | grep 1022, no service run on it. While for {datanode}.log file is not updated.

Initializing secure datanode resources isSaslEnabled:true Opened streaming server at /0.0.0.0:1019 java.net.BindException: Address already in use /0.0.0.0:1022 at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:461) at sun.nio.ch.Net.bind(Net.java:453) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78) at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.getSecureResources(SecureDataNodeStarter.java:158) at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.init(SecureDataNodeStarter.java:94) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:207) Cannot load daemon Service exit with a return value of 3

The error message on Ambari:

resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh -H -E /usr/hdp/3.1.5.0-152/hadoop/bin/hdfs --config /usr/hdp/3.1.5.0-152/hadoop/conf --daemon start datanode' returned 1. ERROR: Cannot set priority of datanode process 7111

Re: Datanode won't start - dfs.datanode.http.address in use

Mahadi — Fri, 13 Aug 2021 04:02:32 GMT

I realize that netstat -lpn | grep 1022 command not display all port so I use netstat -nap | grep 1022 command I found a process run on this port

tcp 0 0 10.10.201.106:1022 10.10.201.10:2049 ESTABLISHED -

Not sure what process it is, seems OS process. So we try to restart the node I after it I find that port 1022 is free. Datanode is successfully started after it.

It is not good to set <1024 for any process, I'll plan to modify all non-standard port on our cluster to the standard one.

question Re: Datanode won't start - dfs.datanode.http.address in use in Support Questions

Datanode won't start - dfs.datanode.http.address in use

Re: Datanode won't start - dfs.datanode.http.address in use