The active NameNode is out of sync with this JournalNode issue.

[#1]

Hello,

Our Cluster has HA configuration using Quorum-based Storage for HDFS. We have to take one of the Journal node out of the cluster and reprovision it. After we put it back in, we got the issues the Active Namenode could not talk to that journal node, but stil be able to talk with other two journal nodes.

I think about disable HA and then reenable HA again, but I don't know if that is a good idea.

Is any one know how to fix this issue?

[#2 · Accepted solution]

I was able to fix the issues by creating missing directory on the Journal host where namenode will write the its' edit files. Make sure the VERSION file is created, otherwise you will get org.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException. 

[#3]

Hi 

can you please specify the dir name on the jornal node?

I am getting no sync with name node and don't know what is missing...

[#4]

Hi reutsc,

I guess it's a bit late, but in case you haven't managed to resolve this, the property that holds this information is called "dfs.journalnode.edits.dir", and it will point to a directory holding a folder named after a logical name of your cluster. So, for example, if the dfs.journalnode.edits.dir is set to /data/jn, the whole path will be /data/jn/xyz-hdp03.

That folder should contain a folder "current", and a file called "in_use.lock", if it is in use. You can also tell by the age of the lock file, and by searching for the journalnode process in your process list. In case you are using Cloudera Manager, you can find the mentioned property in cluster -> HDFS -> Configuration.

Hope this helps,

camypaj

[#5]

after adding the journal node service stop the services

verify for the presence or Please create the folder till the value it configured till the path mentioned under the config -dfs.journalnode.edits.dir

run the scp -r command from other journal node for Clusternamenode folder

Ex: scp -r journal2:/data/hadoop/hdfs/journal/clustername

chown hdfs:hadoop /data/hadoop/hdfs/journal/clustername

Start the Journal node

Observe the Sync of transaction range activity in jounral log and after that edits will be patched as usual