Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

AS NameNode use QJM, why it still writing edits to the local dir while it not use them

Highlighted

AS NameNode use QJM, why it still writing edits to the local dir while it not use them

Explorer

Namenode HA config share edit with QJM, the active NN will write edits to QJM and local directory.

 

When NN starting up, it reads fsimage from local directory, and edits from QJM.

 

As NN is not reading edits from local directory, why it still writing edits to a local directory?

 

4 REPLIES 4

Re: AS NameNode use QJM, why it still writing edits to the local dir while it not use them

Champion
Before an edit is sync'd and committed to the JN it must be written to the local directory. This allows for better recovery as then it fails to commit it still has the edit in the local directory and can try to commit again.

In short better recover as interruptions to 2/3 JNs can happen more frequently than a local write to a disk on the NN.

Re: AS NameNode use QJM, why it still writing edits to the local dir while it not use them

Explorer
"This allows for better recovery as then it fails to commit it still has the edit in the local directory..."

In the fails to commit case, is the failed commit edit exist in memory(double buffer) or not?
If it's still there, then QJM can try to commit again using the edits in memory.

Thanks for your reply, and look for you futher replay.

Re: AS NameNode use QJM, why it still writing edits to the local dir while it not use them

Champion
If the failure is a process or host crash then there isn't anything left in memory to recover from. It is a simple fail safe to ensure that all edits are written to disk as soon as possible.

Re: AS NameNode use QJM, why it still writing edits to the local dir while it not use them

Explorer
Got it, thanks a lot.
Don't have an account?
Coming from Hortonworks? Activate your account here