Moving Journal Node directory from one volume to another volume on the same Host

[#1]

There are two sections which illustrate moving the Journal directory on the same Host.

You can follow one of the section according to your need.

Section 1: ( Service Downtime is not required )

1. Change the settings in Ambari for journal node edits directory(dfs.journalnode.edits.dir) from /hadoop/hdfs/journal/ to /data/1/journal/

2. Don't restart any services immediately.

3. Stop the journal node on NODE1.

a. SSH to NODE1
b. sudo mkdir -p /data/1/journal
c. sudo chown hdfs:hadoop /data/1/journal
d. sudo rsync -ahvAX /hadoop/hdfs/journal/* /data/1/journal/

Start the journal node on NODE1

4. Repeat step 3 for remaining two journal nodes NODE2 and NODE3.

5. Restart the required services accordingly (Rolling or All at once)

Section 2: ( Service Downtime is required )

1. Create the new directory on all Journal nodes.

Example:
# mkdir -p /data/1/journal/

2. Change ownership to "hdfs:hadoop" on all Journal nodes.

Example:
# chown hdfs:hadoop /data/1/journal/

3. Backup an exiting "Fsimage and Edits" from " dfs.namenode.name.dir", "dfs.journalnode.edits.dir" directory on both Namenode and all Journal nodes.

4. Active Namenode - Perform the saveNamespace which will Merge "FSImage and Edits".

# su - hdfs
$ hdfs dfsadmin -safemode enter
$ hdfs dfsadmin -saveNamespace

5. Stop Active Namenode and Standby Namenode.

6. Stop all Journal nodes.

6. Update, latest Journal node directory.

Ambari -> HDFS -> Configs -> Advanced -> Advanced hdfs-site -> dfs.journalnode.edits.dir

7. Start all Journal nodes.

8. Format JN directory.

# su - hdfs
$ hdfs namenode -initializeSharedEdits

Note: The command should be run on one of the Namenode hosts.

9. Start Active Namenode.

Note:

Don't proceed next step until Namenode out of Safemode.

10. Bootstrap Standby Namenode.

# su - hdfs
$ hdfs namenode -bootstrapStandby

Note:

The command should be run on Standby Namenode.

During, You will get the option to format the Storage directory. Press Y and hit enter.

Re-format filesystem in Storage Directory /hadoop/hdfs/namenode ? (Y or N) Y

11. Start Standby Namenode.

[#2]

Thanks @Karthik Palanisamy for sharing this article.

I have done the same activity without initializing shared edits or stopping the entire HDFS service.

Below are the steps which I have followed. Can you please elaborate why we need to re-initialize the shared edits and correct me if the steps I have followed are not appropriate one.

1. Change the settings in Ambari for journal node edits directory(dfs.journalnode.edits.dir) from /hadoop/hdfs/journal/  to /data/1/journal/
2. Don't restart any services immediately.
3. Stop the journal node on NODE1.
   a. SSH to NODE1
   b. sudo mkdir -p /data/1/journal
   c. sudo chown hdfs:hadoop /data/1/journal
   d. sudo rsync -ahvAX /hadoop/hdfs/journal/* /data/1/journal/
   Start the journal node on NODE1
4. Repeat step 3 for remaining two journal nodes NODE2 and NODE3.
5. Restart the required services accordingly (Rolling or All at once)

[#3]

Appreciated @D Giri.

Your step is perfect for changing the Journal directory without affecting any existing service. But my article has steps for doing it offline.

Why we need to re-initialize? I haven't copied the edits and meta from old directory. So there will not be any Journal Layout.

Let me add your steps in another section in the same article.

Thank you for your contribution.