Support Questions
Find answers, ask questions, and share your expertise

Force delete or move services from terminated node?

Solved Go to solution

Force delete or move services from terminated node?

New Contributor

I had a root directory issue on one of my ambari nodes during setup and now I am trying to re-move it from the cluster. When I try to remove the node in the web ui I get a message saying I need to remove the following services: Atlas Metadata Server, DRPC Server, NameNode, Spark History Server, Storm UI Server. My steps were:

1) Decommission node

2) Move the NameNode with the UI

3) Try restart service but received a similar error (without the namenode warning)

4) I tried deleting the node and the services with the API, but that yeilded errors also

Many of these master services do not have a move feature in the UI and I have not seen move operations in the api either. For these remaining services, how can I either force Ambari to delete this terminated node, and then manually install services on new nodes, or move the services to a new node?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Force delete or move services from terminated node?

. @Blair Vanderlugt your Namenode is now running on a different node and the only components left on the "old" node are Atlas Metadata Server, DRPC Server, Spark History Server and Storm UI Server correct?

Unfortunately, I dont think you can delete these components and just reinstall them on a different node, because these are master components. Have you tried deleting the components of this node via API or did you try to delete the complete services via API (service=Spark, components=Spark JHS, Spark Client)

Could you post some of the errors you received when deleting these components/services via API?

Here is what I would try to do:

1) Make a backup of your configuration, databases, etc.; Write down all the config changes you have made to Storm, Atlas, and Spark (master service on your terminated node); this might help https://community.hortonworks.com/questions/4792/a...

2) Delete the whole service including its components from the node, see this https://cwiki.apache.org/confluence/display/AMBARI...

3) Reinstall services on a new node and put the old configuration back in place

View solution in original post

4 REPLIES 4

Re: Force delete or move services from terminated node?

Contributor

@Blair Vanderlugt Can you please let us know the error that you are getting while restarting the service.

If you want to forcefully delete the terminated node, I guess you can delete the host entry in the table "hosts" from the ambari database. I have never tried this. :)

Re: Force delete or move services from terminated node?

Please dont do that, its not a good idea to just delete to entry in the hosts table. Even though modifying the Ambari database is inevitable sometimes, in general I would avoid this way and use the API instead.

Re: Force delete or move services from terminated node?

Contributor

I agree @Jonas Straub. When we have no other choice, we have make changes in the metastore.

Re: Force delete or move services from terminated node?

. @Blair Vanderlugt your Namenode is now running on a different node and the only components left on the "old" node are Atlas Metadata Server, DRPC Server, Spark History Server and Storm UI Server correct?

Unfortunately, I dont think you can delete these components and just reinstall them on a different node, because these are master components. Have you tried deleting the components of this node via API or did you try to delete the complete services via API (service=Spark, components=Spark JHS, Spark Client)

Could you post some of the errors you received when deleting these components/services via API?

Here is what I would try to do:

1) Make a backup of your configuration, databases, etc.; Write down all the config changes you have made to Storm, Atlas, and Spark (master service on your terminated node); this might help https://community.hortonworks.com/questions/4792/a...

2) Delete the whole service including its components from the node, see this https://cwiki.apache.org/confluence/display/AMBARI...

3) Reinstall services on a new node and put the old configuration back in place

View solution in original post