Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Journal Node Sync - moving to new machine

avatar
Rising Star

We have three journal nodes. One was creating issue of sync with name node. Even after multiple rolling restart did not fix sync issue.

 

Later we moved, Journal node services to another node but we again got sync issue with new host’s - journal node to name node.

 

Another things observed here is, location /Data0/dfs/jn directory was empty on new journal node host after moving one journal node service there.

 

Restart of Zookeeper & Journal node did not enable creation of edit logs on new host as well. We have also tried to move everything from working/old JN's location /Data0/dfs/jn/hanameservice/* to new host as per below section.

https://www.cloudera.com/documentation/enterprise/5-4-x/topics/cm_mc_jn.html

 

Can you please suggest, how to enable sync of Journal node with namenode or what step actually we are missing here?

 

CDH - 5.4.+

 

 

Regards,

Kartik

1 ACCEPTED SOLUTION

avatar
Cloudera Employee

Hi Kartik,

Have you tried following these steps:

http://www.cloudera.com/documentation/enterprise/5-4-x/topics/admin_nn_migrate_roles.html?scroll=con...

 

Otherwise, if you can bring down HDFS, you can do the following:

1. Stop HDFS service

2. Copy over JN edits from a (good) JN to the (bad) JN. Make sure you get all edits from all dirs (

dfs.journalnode.edits.dir)
3. Start HDFS service
 
Let me know if that works for you.
 

View solution in original post

5 REPLIES 5

avatar
Cloudera Employee

Hi Kartik,

Have you tried following these steps:

http://www.cloudera.com/documentation/enterprise/5-4-x/topics/admin_nn_migrate_roles.html?scroll=con...

 

Otherwise, if you can bring down HDFS, you can do the following:

1. Stop HDFS service

2. Copy over JN edits from a (good) JN to the (bad) JN. Make sure you get all edits from all dirs (

dfs.journalnode.edits.dir)
3. Start HDFS service
 
Let me know if that works for you.
 

avatar
Explorer

Hi,

I just want to say that we succesfully added two extra JNs to our HA cluster* by following the 3 steps procedure suggested here.

 

For copying content over to new JNs we used rsync, very handy!

 

Regards,
Filippo

 

* running CDH 5.9.1 on centos 6.7 machines

avatar
Contributor

It seems really strange that the process of adding in a new JournalNode doesn't DO THIS FOR YOU. Basically, if you change or add journalnodes to your setup, nothing will work until/unless you do this behind the scenes stuff. WHY?

avatar
New Contributor

I echo on the same question. Why would we need to copy all the contents from the running journal nodes to the new nodes. even though when the journal service on the new nodes is started and entry marked in the configuration of the namenode, it should automatically get sync'ed up all the edits.

 

WHY do we copy the old edits to new journal servers?

 

avatar
Rising Star

Hello,

 

I have three Journal Nodes, I am facing same issue for 2 Journal Nodes. So i have copied the JournalEdits from good JN to bad JN like two other JN's and restarted.

But still i am seeing same sync issue and can see below errors.

 Cannot locate configuration: tried hadoop-metrics2-journalnode.properties,hadoop-metrics2.properties

 

ERROR org.apache.hadoop.hdfs.qjournal.server.JournalNode RECEIVED SIGNAL 15: SIGTERM
WARN org.apache.hadoop.metrics2.impl.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-journalnode.properties,hadoop-metrics2.properties

 

Thanks in advance....!!