Hello, After a mass disk operation on our test environment, we have lost all the data in /data dir which was assigned as storage directory for Zookeeper, Hadoop and Falcon (the list yet we know) Since it was our test cluster, data is not important but I don't want to reinstall all the components. I also want to learn how to recover the cluster running from this state. In /data dir we only have folders but no files. After struggling a little on ZKFailoverController, I was able to start it with -formatZK flag. Now however, I am unable to start namenode(s) getting below exception: 10.0.109.12:8485: Directory /hadoop/hdfs/journal/testnamespace is in an inconsistent state: Can't format the storage directory because the current directory is not empty. I have tried; - removing lost+found folder on mount root, - changing ownership of all folders under /data/hadoop/hdfs to hdfs:hadoop - changing permission of all folders to 777 /data/hadoop/hdfs PS: I have updated ownership of path /hadoop/hdfs/ which contains journal folder and it led me to move one step forward: 17/12/01 14:20:26 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Cannot remove current directory: /data/hadoop/hdfs/namenode/current PS: I have removed contents of /data/hadoop/hdfs/namenode/current and now it keeps checking 8485 ports of all Journal quorum nodes. 17/12/01 16:04:35 INFO ipc.Client: Retrying connect to server: bigdata2/10.0.109.11:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) and keeps printing below line in hadoop-hdfs-zkfc-bigdata2.out file Proceed formatting /hadoop-ha/testnamespace? (Y or N) Invalid input: Do you have any suggestion? Or should I give up?
... View more
