Reply
Explorer
Posts: 20
Registered: ‎11-29-2016

AS NameNode use QJM, why it still writing edits to the local dir while it not use them

Namenode HA config share edit with QJM, the active NN will write edits to QJM and local directory.

 

When NN starting up, it reads fsimage from local directory, and edits from QJM.

 

As NN is not reading edits from local directory, why it still writing edits to a local directory?

 

Posts: 563
Topics: 3
Kudos: 78
Solutions: 49
Registered: ‎08-16-2016

Re: AS NameNode use QJM, why it still writing edits to the local dir while it not use them

Before an edit is sync'd and committed to the JN it must be written to the local directory. This allows for better recovery as then it fails to commit it still has the edit in the local directory and can try to commit again.

In short better recover as interruptions to 2/3 JNs can happen more frequently than a local write to a disk on the NN.
Explorer
Posts: 20
Registered: ‎11-29-2016

Re: AS NameNode use QJM, why it still writing edits to the local dir while it not use them

"This allows for better recovery as then it fails to commit it still has the edit in the local directory..."

In the fails to commit case, is the failed commit edit exist in memory(double buffer) or not?
If it's still there, then QJM can try to commit again using the edits in memory.

Thanks for your reply, and look for you futher replay.
Posts: 563
Topics: 3
Kudos: 78
Solutions: 49
Registered: ‎08-16-2016

Re: AS NameNode use QJM, why it still writing edits to the local dir while it not use them

If the failure is a process or host crash then there isn't anything left in memory to recover from. It is a simple fail safe to ensure that all edits are written to disk as soon as possible.
Explorer
Posts: 20
Registered: ‎11-29-2016

Re: AS NameNode use QJM, why it still writing edits to the local dir while it not use them

Got it, thanks a lot.
Announcements