Process for Moving Journal Nodes from One Host to Another

[#1]

In case you would like to move JournalNode service to another host, here are the steps to do so: 1. Put HDFS in safemode

su - hdfs -c 'hdfs dfsadmin -fs hdfs://<active node>:8020 -safemode enter'

2. Execute a save namespace of the Active NameNode

su - hdfs -c 'hdfs dfsadmin -fs hdfs://<active node>:8020 -saveNamespace'

3. Stop all services via Ambari

4. Add journal node to the Ambari database through the API call:

export AMBARI_USER=admin
export PASSWORD=admin
export AMBARI_HOST=localhost
export CLUSTER_NAME=cluster
export MOVE_FROM='old-host.abcd.com'
export MOVE_TO='new-host.abcd.com'


# Tell to ambari we want to install this component on new_host
curl -u $AMBARI_USER:$AMBARI_PASSWORD -H "X-Requested-By:ambari" -i -X POST http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/hosts/$MOVE_TO/host_components/JOURNALNODE

OR

curl
 -v -u admin:admin -H "X-Requested-By:ambari" -i -X POST -d 
'{"host_components" : [{"HostRoles":{"component_name":"JOURNALNODE"}}] 
}' 
http://<ambari-server>:8080/api/v1/clusters/<clustername>/hosts?Hosts/host_name=<new
 journal node hostname> 

5. Within the Ambari web UI, click on the "Host" tab to display all the hosts for the cluster. Click on the host in which you just added the JournalNode service to in step 4. On this page you should see the JournalNode component in the "Components" section. Next to the JournalNode label is a drop down button, click it and select the "install" option. A progress bar should show up displaying the progress of the installation.

OR via Curl

# Trigger installation

curl -u $AMBARI_USER:$AMBARI_PASSWORD -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context": "Install JournalNode","query":"HostRoles/component_name.in('JOURNALNODE')"}, "Body":{"HostRoles": {"state": "INSTALLED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/hosts/$MOVE_TO/host_components

6. Once the installation finishes, update the configuration property below within in the Ambari UI within the HDFS section.

<dfs.namenode.shared.edits.dir> 

to include the new journal node and exclude the old one

7. Start the new installed journal node

or Via Curl

# Start JournalNode on new host 
curl -u $AMBARI_USER:$AMBARI_PASSWORD -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context": "Start JournalNode","query":"HostRoles/component_name.in('JOURNALNODE')"}, "Body":{"HostRoles": {"state": "STARTED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/hosts/$MOVE_TO/host_components/JOURNALNODE

8. Start the rest of the journal nodes

9. Start the active NameNode to refresh the configuration and then stop it.

10. On the previous active node, run

hdfs namenode -initializeSharedEdits -force" 

11. Restart the active NameNode

12. On the standby NameNode run

"hdfs namenode -bootstrapStandby" 

(it will prompt you to “re-format filesystem in storage directory /hadoop/hdfs/namenode? (Y or N) , answer Y “ )

13. Start DataNodes

14. Start standby NameNode

15. Remove journal node by the following api call

# Stop JournalNode on old host 
curl -u $AMBARI_USER:$AMBARI_PASSWORD -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context": "Stop JournalNode","query":"HostRoles/component_name.in('JOURNALNODE')"}, "Body":{"HostRoles": {"state": "INSTALLED"}}}' http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/hosts/$MOVE_FROM/host_components/JOURNALNODE


# Remove the old component
curl -u $AMBARI_USER:$AMBARI_PASSWORD -H "X-Requested-By:ambari" -i -X DELETE http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/hosts/$MOVE_FROM/host_components/JOURNALNODE

16. Start all services (ENSURE THIS IS DONE AND ZOOKEEPER IS UP AND RUNNING OR NEITHER NAMENODE WILL BE ACTIVE)

(Reference support note also)

[#2]

Thanks Ancil, when we followed these steps we were able to move the journal node but for some reason it did not get initialized properly. Looks like one step is missing, either copy over directory for the edit files from old node to new node OR copy VERSION file to /hadoop/hdfs/journal/<clustername>/current before starting the journal node

[#3]

HI!

What if my current journalNode went down due to bad disk and re-setup the machine from the start.

What steps do I need to follow? Thank you!

[#4]

Note that Manage Journal Node is now available through UI in Ambari 2.5.1

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-operations/content/manage_journal...

[#5]

the step 3. Stop all services via Ambari what means . Al the services of the cluster, including by example kafka, spark etc or only all the services of service hdfs. I need know.

[#6]

Hello, 

Do you have any test to be sure the cluster is working fine after all those steps ? 

I move one node ever 3. 

I put some files on the hdfs and i don't see file system in the dfs.journalnode.edits.dir even whether for the old or new Journal Node. 

Best Regards

Abdou