Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

The active NameNode is out of sync with this JournalNode issue.

avatar
Rising Star

Hello,

Our Cluster has HA configuration using Quorum-based Storage for HDFS. We have to take one of the Journal node out of the cluster and reprovision it. After we put it back in, we got the issues the Active Namenode could not talk to that journal node, but stil be able to talk with other two journal nodes.

I think about disable HA and then reenable HA again, but I don't know if that is a good idea.

Is any one know how to fix this issue?

1 ACCEPTED SOLUTION

avatar
Rising Star

I was able to fix the issues by creating missing directory on the Journal host where namenode will write the its' edit files. Make sure the VERSION file is created, otherwise you will get org.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException. 

View solution in original post

4 REPLIES 4

avatar
Rising Star

I was able to fix the issues by creating missing directory on the Journal host where namenode will write the its' edit files. Make sure the VERSION file is created, otherwise you will get org.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException. 

avatar
Explorer

Hi 

 

can you please specify the dir name on the jornal node?

I am getting no sync with name node and don't know what is missing...

avatar
Rising Star

Hi reutsc,

 

I guess it's a bit late, but in case you haven't managed to resolve this, the property that holds this information is called "dfs.journalnode.edits.dir", and it will point to a directory holding a folder named after a logical name of your cluster. So, for example, if the dfs.journalnode.edits.dir is set to /data/jn, the whole path will be /data/jn/xyz-hdp03.

That folder should contain a folder "current", and a file called "in_use.lock", if it is in use. You can also tell by the age of the lock file, and by searching for the journalnode process in your process list. In case you are using Cloudera Manager, you can find the mentioned property in cluster -> HDFS -> Configuration.

Hope this helps,

camypaj

avatar
New Contributor

after adding the journal node service stop the services

verify for the presence or Please create the folder till the value it configured till the path mentioned under the config -dfs.journalnode.edits.dir

run the scp -r command from other journal node for Clusternamenode folder

 

Ex: scp -r journal2:/data/hadoop/hdfs/journal/clustername

chown hdfs:hadoop /data/hadoop/hdfs/journal/clustername

Start the Journal node

Observe the Sync of transaction range activity in jounral log and after that edits will be patched as usual