We tried to restart the namenode with the Unknown state without success. We are getting the following error in the logs: 2022-01-16 06:46:13,099 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode. java.net.BindException: Port in use: xx-xxx-x2-xx01.xxxxx.xx:50470 at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1197) at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1219) at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1278) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1133) at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:177) at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:869) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:691) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:317) at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1184) at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1215) ... 9 more 2022-01-16 06:46:13,101 INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: java.net.BindException: Port in use: xx-xxx-x2-xx01.xxxxx.xx:50470 2022-01-16 06:46:13,102 INFO namenode.NameNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at xx-xxx-x2-xx01.xxxxx.xx/xx.x.xx.xx   We runned th command lsof -i:50470 on the host to see what process is running on this port we got the following result: java 5356 hdfs 354u IPv4 4207102955 0t0 TCP xx-xxx-x2-xx01.xxxxx.xx:56788->xx-xxx-x2-xx01.xxxxx.xx:50470 (ESTABLISHED)   Do we need to kill that process by running kill 5356 and than try to start the namenode again? ... View more

I have executed the command "su -l hdfs -c "/usr/hdp/current/hadoop-hdfs-namenode/../hadoop/sbin/hadoop-daemon.sh start namenode" i have the following warning on the command line: WARNING: Use of this script to start HDFS daemons is deprecated. WARNING: Attempting to execute replacement "hdfs --daemon start" instead.   and the following error on the log: 2021-12-19 14:06:55,554 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode. org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 274473528 at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:226) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:160) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:890) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1090) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 274473527; expected file to go up to 274474058 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:197) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:179) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:213) ... 12 more 2021-12-19 14:06:55,557 INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0. Expected transaction ID was 274473528 2021-12-19 14:06:55,558 INFO namenode.NameNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at XX-XXX-XX-XXXX.XXXXX.XX/XX.X.XX.XX  One thing, went to check the 3 host that have the journal nodes (nn1, nn2, host3). I i did the following command: cd /hadoop/hdfs/journal/<Cluster_name>/current ll | wc -l 9653 they all have the same amount of files.   ... View more

May i contact you directly regarind this namenode issue if you are available?   Thank you ... View more

I will try the command ... View more

@Sheltoni have tried to do the first step (hdfs dfsadmin -safemode enter) but i am getting the following error:     safemode: Call From XX-XXX-XX-XXXX.XXXXX.XX/XX.X.XX.XX to XX-XXX-XX-XXXX.XXXXX.XX:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused     ... View more

HI @Shelton The thing is from the ambari UI console we have the following view: As you can see both namenode are down is there a way to start them back? ... View more

According to the log namenode2 is not able to start because of that: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 274488048; expected file to go up to 274488109 Any idea on how to fix this? ... View more

Could you please give me the steps that need to be follow in case the fs image and edit log are corrup in HA setup (2 namenodes)? I am still note able to start both namenode.    Thank you ... View more