Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

shared edits dir question

shared edits dir question

Explorer

Hello,

I have a question regarding "shared edits dir" location. It is in /hadoop/hdfs/journal/clustername/current/

it is stored in hdfs file system. Is it suppose to work or can we configure in any other file sytems like NFS?

3 REPLIES 3
Highlighted

Re: shared edits dir question

Mentor

@Mokkan Mok

The shared edits dir is not stored directory in HDFS to validate that run

$hdfs dfs -ls /hadoop 

it will throw an error because HDFS mount point doesn't exist. The "shared edits dir" location is created When you run the format command for HDFS, it is preparing NameNode fsimage file so that it knows where all of the storage blocks are across the data disks and hadoop mounts its own filesystem using chroot.

The /hadoop directory will ONLY be available after starting the cluster as its invisible to the normal Linux command like

$ df -h 

NFS

Standby node to keep its state synchronized with the Active node,two nodes both have access to a directory on a shared storage device i.e NFS.Currently only a single shared edits directory is supported,so the availability of the Cluster is limited by the availability of this shared edits directory in order to remove SPOF, you MUST ensure multiple network paths to the NFS storage, and redundancy in the storage itself (disk, network, and power)

It's strongly recommended to soft mount the NFS directory and periodically making a backup of the NN's fsimage and JN's edits files to ensure your BCP operational procedures are rock solid & tested when it comes to the NameNode and its fsimage/edits files recovery

Highlighted

Re: shared edits dir question

Explorer

Hi Geoffery,

Does all the journal node manages their own edit log in their local disk ?

Highlighted

Re: shared edits dir question

Mentor

@Mokkan Mok

I have attached a visual that will help you understand, its already fault tolerant, so basically, you have 1 copy on three different local disks, so if you lost one set of JornalNode disk you still shoule be able to recover !


qjm-arch.jpg
Don't have an account?
Coming from Hortonworks? Activate your account here