Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Process for moving HDP Services manually

Solved Go to solution
Highlighted

Process for moving HDP Services manually

New Contributor

We are looking for a documented process for moving services that are currently not supported by Ambari. The Services I am currently looking for information for:

1. HDFS Jornal Node

2. Map Reduce Job History Server

3. Spark History Server

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Process for moving HDP Services manually

If there is no existing documentation covering the JournalNode, then here is my recommendation.

  1. Bootstrap the new server by copying the contents of dfs.journalnode.edits.dir from an existing JournalNode.
  2. Start JournalNode on the new server.
  3. Reconfigure the NameNodes to include the new server in dfs.namenode.shared.edits.dir.
  4. Restart standby NN and verify it remains healthy.
  5. Restart active NN and verify it remains healthy.
  6. Reconfigure the NameNodes to remove the old server from dfs.namenode.shared.edits.dir.
  7. Restart standby NN and verify it remains healthy.
  8. Restart active NN and verify it remains healthy.

Some might note that during the copy in step 1, it's possible that additional transactions are being logged concurrently, so the copy might be out-of-date immediately. This is not a problem though. The JournalNode is capable of "catching up" by synchronizing data from other running JournalNodes. In fact, step 1 is really just an optimization of this "catching up".

6 REPLIES 6

Re: Process for moving HDP Services manually

If there is no existing documentation covering the JournalNode, then here is my recommendation.

  1. Bootstrap the new server by copying the contents of dfs.journalnode.edits.dir from an existing JournalNode.
  2. Start JournalNode on the new server.
  3. Reconfigure the NameNodes to include the new server in dfs.namenode.shared.edits.dir.
  4. Restart standby NN and verify it remains healthy.
  5. Restart active NN and verify it remains healthy.
  6. Reconfigure the NameNodes to remove the old server from dfs.namenode.shared.edits.dir.
  7. Restart standby NN and verify it remains healthy.
  8. Restart active NN and verify it remains healthy.

Some might note that during the copy in step 1, it's possible that additional transactions are being logged concurrently, so the copy might be out-of-date immediately. This is not a problem though. The JournalNode is capable of "catching up" by synchronizing data from other running JournalNodes. In fact, step 1 is really just an optimization of this "catching up".

Re: Process for moving HDP Services manually

Re: Process for moving HDP Services manually

For Spark, there is no explicit move Spark history server feature in Ambari.

Just use the Ambari REST API to delete Spark component and re-install Spark History server onto a new node.

Re: Process for moving HDP Services manually

New Contributor

@vshukla can you provide details on REST API to be used ?

Re: Process for moving HDP Services manually

Rising Star

@Vinod : Here are the API details:

Delete SPARK Job Server:

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/services/SPARK
curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop Component"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/hosts/<old-host>/host_components/SPARK_JOBHISTORYSERVER
curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/services/SPARK/components/SPARK_JOBHISTORYSERVER

Add SPARK Job Server to New Server

curl -u admin:admin -H "X-Requested-By: ambari" -i -X POST http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/services/SPARK/components/SPARK_JOBHISTORYSERVER
curl -u admin:admin -H "X-Requested-By: ambari" -i -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"SPARK_JOBHISTORYSERVER"}}] }' http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/hosts?Hosts/host_name=<new-host>
curl -u admin:admin -H "X-Requested-By: ambari" -i -X PUT -d '{"ServiceInfo": {"state" : "INSTALLED"}}' http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/services/SPARK

Re: Process for moving HDP Services manually

New Contributor

Answer from Ambari team for MapReduce History Server

Delete Mapreduce History Server if server died

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/services/MAPREDUCE2

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop Component"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/hosts/<old-host-name>/host_component...

curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/services/MAPREDUCE2/components/HISTO...

Add Mapreduce History Server to New Server

curl -u admin:admin -H "X-Requested-By: ambari" -i -X POST http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/services/MAPREDUCE2/components/HISTO...

curl -u admin:admin -H "X-Requested-By: ambari" -i -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"HISTORYSERVER"}}] }' http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/hosts?Hosts/host_name=<new-host-name...;

curl -u admin:admin -H "X-Requested-By: ambari" -i -X PUT -d '{"ServiceInfo": {"state" : "INSTALLED"}}' http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/services/MAPREDUCE2

Go to Ambari Web > Services > MapReduce2 > Configs > Advanced, and modify mapreduce.jobhistory.address and mapreduce.jobhistory.webapp.address to point to the new History Server host.

Restart MapReduce2 Service after saving the configuration.