Process for moving HDP Services manually

[#1]

We are looking for a documented process for moving services that are currently not supported by Ambari. The Services I am currently looking for information for:

1. HDFS Jornal Node

2. Map Reduce Job History Server

3. Spark History Server

[#2 · Accepted solution]

If there is no existing documentation covering the JournalNode, then here is my recommendation.

1. Bootstrap the new server by copying the contents of dfs.journalnode.edits.dir from an existing JournalNode.
2. Start JournalNode on the new server.
3. Reconfigure the NameNodes to include the new server in dfs.namenode.shared.edits.dir.
4. Restart standby NN and verify it remains healthy.
5. Restart active NN and verify it remains healthy.
6. Reconfigure the NameNodes to remove the old server from dfs.namenode.shared.edits.dir.
7. Restart standby NN and verify it remains healthy.
8. Restart active NN and verify it remains healthy.

Some might note that during the copy in step 1, it's possible that additional transactions are being logged concurrently, so the copy might be out-of-date immediately. This is not a problem though. The JournalNode is capable of "catching up" by synchronizing data from other running JournalNodes. In fact, step 1 is really just an optimization of this "catching up".

[#3]

See also

https://community.hortonworks.com/articles/14612/process-for-moving-journal-nodes-from-one-host-to.h...

[#4]

For Spark, there is no explicit move Spark history server feature in Ambari.

Just use the Ambari REST API to delete Spark component and re-install Spark History server onto a new node.

[#5]

@vshukla can you provide details on REST API to be used ?

[#6]

@Vinod : Here are the API details:

Delete SPARK Job Server:

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/services/SPARK
curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop Component"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/hosts/<old-host>/host_components/SPARK_JOBHISTORYSERVER
curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/services/SPARK/components/SPARK_JOBHISTORYSERVER

Add SPARK Job Server to New Server

curl -u admin:admin -H "X-Requested-By: ambari" -i -X POST http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/services/SPARK/components/SPARK_JOBHISTORYSERVER
curl -u admin:admin -H "X-Requested-By: ambari" -i -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"SPARK_JOBHISTORYSERVER"}}] }' http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/hosts?Hosts/host_name=<new-host>
curl -u admin:admin -H "X-Requested-By: ambari" -i -X PUT -d '{"ServiceInfo": {"state" : "INSTALLED"}}' http://:8080/api/v1/clusters/clustername/services/SPARK>:8080/api/v1/clusters/clustername/services/SPARK

[#7]

Answer from Ambari team for MapReduce History Server

Delete Mapreduce History Server if server died

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop Service"},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/services/MAPREDUCE2

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop Component"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/hosts/<old-host-name>/host_component...

curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/services/MAPREDUCE2/components/HISTO...

Add Mapreduce History Server to New Server

curl -u admin:admin -H "X-Requested-By: ambari" -i -X POST http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/services/MAPREDUCE2/components/HISTO...

curl -u admin:admin -H "X-Requested-By: ambari" -i -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"HISTORYSERVER"}}] }' http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/hosts?Hosts/host_name=<new-host-name...;

curl -u admin:admin -H "X-Requested-By: ambari" -i -X PUT -d '{"ServiceInfo": {"state" : "INSTALLED"}}' http://<ambari-server-name>:8080/api/v1/clusters/<cluster-name>/services/MAPREDUCE2

Go to Ambari Web > Services > MapReduce2 > Configs > Advanced, and modify mapreduce.jobhistory.address and mapreduce.jobhistory.webapp.address to point to the new History Server host.

Restart MapReduce2 Service after saving the configuration.