Datanode won't start - dfs.datanode.http.address in use

[#1]

One of of my data-node won't start. The error log on {HADOOP_LOG}/hdfs/{datanode}.err shown that the port of dfs.datanode.http.address already in use. We use custom port for  dfs.datanode.http.address. When I run netstat -lpn | grep 1022, no service run on it. While for {datanode}.log file is not updated.

Initializing secure datanode resources
isSaslEnabled:true
Opened streaming server at /0.0.0.0:1019
java.net.BindException: Address already in use /0.0.0.0:1022
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:461)
        at sun.nio.ch.Net.bind(Net.java:453)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78)
        at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.getSecureResources(SecureDataNodeStarter.java:158)
        at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.init(SecureDataNodeStarter.java:94)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:207)
Cannot load daemon
Service exit with a return value of 3

The error message on Ambari:

resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh  -H -E /usr/hdp/3.1.5.0-152/hadoop/bin/hdfs --config /usr/hdp/3.1.5.0-152/hadoop/conf --daemon start datanode' returned 1. ERROR: Cannot set priority of datanode process 7111

[#2 · Accepted solution]

I realize that netstat -lpn | grep 1022 command not display all port so I use netstat -nap | grep 1022 command I found a process run on this port

tcp 0 0 10.10.201.106:1022 10.10.201.10:2049 ESTABLISHED -

Not sure what process it is, seems OS process. So we try to restart the node I after it I find that port 1022 is free. Datanode is successfully started after it.

It is not good to set <1024 for any process, I'll plan to modify all non-standard port on our cluster to the standard one.