Cluster regain Help: One of my Namenode directory lost (means two directories are kept for name node back up in that one node has lost).

[#1]

Second,Ambari was not able to start because of /var/log got filled. Third Connection exception is giving when we try to type hadoop command. we want to regain back the cluster any suggestions to regain steps manually since Ambari is not working

[#2 · Accepted solution]

1. NN data lost.

Is it that the disk with NN directories crashed or have you deleted them? Is this with HA or non-HA. With non-HA, if both data directories of NN have no data, then you will run into data loss issues. You can revive and get to some state from secondary NN data directories but can not guarantee no data loss. If there is no useful data, you can always issue a NN format and start fresh. (You will need to manually update tez and mapreduce apps, you can get the information from manual install documentation)

2. Ambari not starting.

Clean up /var/log and start it back.

3. Most likely HDFS services are not up. Filled up disk can kill processes. Once ambari is up, see which one is running and which one is not. Againt if NN data is lost, then NN will not start up.