Force delete or move services from terminated node?

[#1]

I had a root directory issue on one of my ambari nodes during setup and now I am trying to re-move it from the cluster. When I try to remove the node in the web ui I get a message saying I need to remove the following services: Atlas Metadata Server, DRPC Server, NameNode, Spark History Server, Storm UI Server. My steps were:

1) Decommission node

2) Move the NameNode with the UI

3) Try restart service but received a similar error (without the namenode warning)

4) I tried deleting the node and the services with the API, but that yeilded errors also

Many of these master services do not have a move feature in the UI and I have not seen move operations in the api either. For these remaining services, how can I either force Ambari to delete this terminated node, and then manually install services on new nodes, or move the services to a new node?

[#2]

@Blair Vanderlugt Can you please let us know the error that you are getting while restarting the service.

If you want to forcefully delete the terminated node, I guess you can delete the host entry in the table "hosts" from the ambari database. I have never tried this. 🙂

[#3]

Please dont do that, its not a good idea to just delete to entry in the hosts table. Even though modifying the Ambari database is inevitable sometimes, in general I would avoid this way and use the API instead.

[#4]

I agree @Jonas Straub. When we have no other choice, we have make changes in the metastore.

[#5 · Accepted solution]

. @Blair Vanderlugt your Namenode is now running on a different node and the only components left on the "old" node are Atlas Metadata Server, DRPC Server, Spark History Server and Storm UI Server correct?

Unfortunately, I dont think you can delete these components and just reinstall them on a different node, because these are master components. Have you tried deleting the components of this node via API or did you try to delete the complete services via API (service=Spark, components=Spark JHS, Spark Client)

Could you post some of the errors you received when deleting these components/services via API?

Here is what I would try to do:

1) Make a backup of your configuration, databases, etc.; Write down all the config changes you have made to Storm, Atlas, and Spark (master service on your terminated node); this might help https://community.hortonworks.com/questions/4792/a...

2) Delete the whole service including its components from the node, see this https://cwiki.apache.org/confluence/display/AMBARI...

3) Reinstall services on a new node and put the old configuration back in place