Created 08-12-2021 05:23 PM
One of of my data-node won't start. The error log on {HADOOP_LOG}/hdfs/{datanode}.err shown that the port of dfs.datanode.http.address already in use. We use custom port for dfs.datanode.http.address. When I run netstat -lpn | grep 1022, no service run on it. While for {datanode}.log file is not updated.
Initializing secure datanode resources
isSaslEnabled:true
Opened streaming server at /0.0.0.0:1019
java.net.BindException: Address already in use /0.0.0.0:1022
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:461)
at sun.nio.ch.Net.bind(Net.java:453)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78)
at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.getSecureResources(SecureDataNodeStarter.java:158)
at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.init(SecureDataNodeStarter.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:207)
Cannot load daemon
Service exit with a return value of 3
The error message on Ambari:
resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh -H -E /usr/hdp/3.1.5.0-152/hadoop/bin/hdfs --config /usr/hdp/3.1.5.0-152/hadoop/conf --daemon start datanode' returned 1. ERROR: Cannot set priority of datanode process 7111
Created on 08-12-2021 08:47 PM - edited 08-12-2021 09:02 PM
I realize that netstat -lpn | grep 1022 command not display all port so I use netstat -nap | grep 1022 command I found a process run on this port
tcp 0 0 10.10.201.106:1022 10.10.201.10:2049 ESTABLISHED -
Not sure what process it is, seems OS process. So we try to restart the node I after it I find that port 1022 is free. Datanode is successfully started after it.
It is not good to set <1024 for any process, I'll plan to modify all non-standard port on our cluster to the standard one.
Created on 08-12-2021 08:47 PM - edited 08-12-2021 09:02 PM
I realize that netstat -lpn | grep 1022 command not display all port so I use netstat -nap | grep 1022 command I found a process run on this port
tcp 0 0 10.10.201.106:1022 10.10.201.10:2049 ESTABLISHED -
Not sure what process it is, seems OS process. So we try to restart the node I after it I find that port 1022 is free. Datanode is successfully started after it.
It is not good to set <1024 for any process, I'll plan to modify all non-standard port on our cluster to the standard one.