shared edits dir question

[#1]

Hello,

I have a question regarding "shared edits dir" location. It is in /hadoop/hdfs/journal/clustername/current/

it is stored in hdfs file system. Is it suppose to work or can we configure in any other file sytems like NFS?

[#2]

@Mokkan Mok

The shared edits dir is not stored directory in HDFS to validate that run

$hdfs dfs -ls /hadoop 

it will throw an error because HDFS mount point doesn't exist. The "shared edits dir" location is created When you run the format command for HDFS, it is preparing NameNode fsimage file so that it knows where all of the storage blocks are across the data disks and hadoop mounts its own filesystem using chroot.

The /hadoop directory will ONLY be available after starting the cluster as its invisible to the normal Linux command like

$ df -h 

NFS

Standby node to keep its state synchronized with the Active node,two nodes both have access to a directory on a shared storage device i.e NFS.Currently only a single shared edits directory is supported,so the availability of the Cluster is limited by the availability of this shared edits directory in order to remove SPOF, you MUST ensure multiple network paths to the NFS storage, and redundancy in the storage itself (disk, network, and power)

It's strongly recommended to soft mount the NFS directory and periodically making a backup of the NN's fsimage and JN's edits files to ensure your BCP operational procedures are rock solid & tested when it comes to the NameNode and its fsimage/edits files recovery

[#3]

Hi Geoffery,

Does all the journal node manages their own edit log in their local disk ?

[#4]

@Mokkan Mok

I have attached a visual that will help you understand, its already fault tolerant, so basically, you have 1 copy on three different local disks, so if you lost one set of JornalNode disk you still shoule be able to recover !