Thank you @rki_ ! That is absolutely what happened. I had a node that the /tmp/ folder still contained old journalnode data. After cleaning it up and doing initializeSharedEdits i managed to start cluster.  Note: I had this exact exception on two slave nodes: WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage java.io.IOException: There appears to be a gap in the edit log. We expected txid 121994, but got txid 121998. I did hdfs namenode -recover on both slave nodes and then was able to start both namenodes propely. The data is replicated within all 3 nodes.  Thank you so much for the help! ... View more

Thank you @rki_ seems like a takeaway is to never format namenode if you have data on the cluster. Before starting the whole process of upgrading debian version of the machines in the cluster i copied all the data from master node into another single node hadoop cluster running on seperate machine.  Now I used that datafile to build a new single node cluster that is started succesfully. But now I want to extend the sinlge node cluster to be multi node cluster with 3 nodes.  I stopped the sinlge node cluster and edit all the configuration files on all three nodes following the setup I had previously. This is how the configuration files look like: https://codefile.io/f/jsklFb1h0M   I also copied the /hdfs/dfs/name to the other two nodes without the /hdfs/dfs/data. I want to let the hadoop handle by auto replicating the data from the master node. I can successfully start zookeper service: #start the zookeeper /usr/local/zookeeper/bin/zkServer.sh start And also start the journalnode: #start journalnode hadoop/bin/hdfs --daemon start journalnode The usual next step would be: hadoop/bin/hdfs namenode -initializeSharedEdits But that will prompt me with: Re-format filesystem in QJM to [vm-ip1:8485, vm-ip2:8485, vm-ip3:8485] ? (Y or N) I guess if I reformat the filesystem again i will lose the data on the master node again. How can I start this multi node cluster and replicate the data on other two nodes succesfully with the described state? ... View more

@rki_  Is there any way to recover the data if the NameNode was not backed up. I had a 3 node cluster. On each machine I now have a datafile with 11G of data on it: 11G ./BP-1475975986-127.0.0.1-1447139386120/current/finalized/subdir0 I want to save the data if possible, is there a way to start a new single node cluster and then map this block data with the new cluster or I can assume my data is lost?  Basically what Im asking is when the namenode metadata is modified by the hdfs reformat command, the datafile itself is unusable or there is a way to get it back? ... View more

Thank you @VidyaSargur! Hope to hear from them soon! ... View more