question Re: Ambari Mysql Database lost in Support Questions

Ambari Mysql Database lost

moubaba — Wed, 30 Oct 2019 17:40:12 GMT

Hello everyone

I have development cluster where I lost my ambari database which was on a mysql database.

Is there anyway to reinstall ambari server and have my cluster back up again ?

Thank you for your help

Re: Ambari Mysql Database lost

Shelton — Wed, 30 Oct 2019 19:35:06 GMT

@moubaba

I know its a dompting task but unfortunately, I could create a link by an article for Jordan Moore let me try to elaborate.

Simple advice always run a crontab job to dump all your databases or at leat the ambari DB nightly !

Firstly Ambari doesn't control any data residing on the data nodes, so you should be safe there. Stop all ambari-agents in the cluster, maybe even uninstalling it so as to let all the Hadoop components remain running "in the dark".

Install and set up a new Ambari server, add a cluster but do not register any hosts [very important]. If you uninstalled the ambari -agents on the nodes please do re-install ensure the ambari-server and the agents are the same versions.

Do the below if you removed Ambari and the agents

# yum repolist

On the ambari-server

# yum install ambari-server

# yum install ambari-agent

On the other hosts

# yum install ambari-agent

Reconfigure manually the ambari agents [ambari-agent.ini] to point at the new Ambari server address, and start them.

Add the hosts in the Ambari server UI, selecting "manual registration" option the hosts register successfully since the agents are running a point to note install and configure an ambari-agent on the Ambari server to point to itself !!!!

After this, you get the option of installing clients and servers.

Now, you could try to "reinstall" what is already there, but you might want to deselect all the servers on the data node column.

In theory, it will try to perform OS package installation, and say that the service already exists, and doesn't error out. If it does error, then restart the install process, but deselect everything. At which point, it should continue and now you have Ambari back up and running with all the hosts monitored, just with no processes to configure.

To add the services back, you would need to use the Ambari REST API to add back in the respective Services, Components, and Host Components that you have running on the cluster. If you can't remember all the Services, Components, and Host Components then go to each host and do a process check to see what's running.

Variables

export AMBARI_USER=admin
export AMBARI_PASSWD=admin
export CLUSTER_NAME=<New_Cluster_name>
export AMBARI_SERVER=<Your_new_ambari_server_FQDN>
export AMBARI_PORT=8080

List all services related to hive

In the below example I am listing all hive components

curl -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X GET "http://$AMBARI_SERVER:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/services/HIVE"

Adding back pig

curl -k -u $AMBARI_USER:$AMBARI_PASSWD -H "X-Requested-By:ambari" -i -X POST -d '{"RequestInfo":{"context":"Install PIG"}, "Body":{"HostRoles":{"state":"INSTALLED"}}}' 'http://'$AMBARI_SERVER':$AMBARI_PORT/api/v1/clusters/'$CLUSTER_NAME'/services/PIG/components/PIG'

This is the time to get your hands dirty 🙂 This isn't production so it should train you for a real situation
Happy hadooping !!

Re: Ambari Mysql Database lost

moubaba — Thu, 31 Oct 2019 08:16:20 GMT

Thanks Shelton for your very detailled response. It is really helpful.

Concerning the adding back services with the API : does it install the services again ?

For example, if I add back Hive services does it install hive again ?

Re: Ambari Mysql Database lost

moubaba — Thu, 31 Oct 2019 08:35:35 GMT

I got another question also.

What about the views ? Like File View and Hive View, can I create them back ?

Re: Ambari Mysql Database lost

moubaba — Thu, 31 Oct 2019 15:50:11 GMT

Hello Shelton

Any help to recreate Ambari Views ?

Re: Ambari Mysql Database lost

Shelton — Thu, 31 Oct 2019 20:45:38 GMT

@moubaba

Here is a fantastic document by Artem Ervits Ambari Views REST API Overview

Hope that answers your query

Happy hadooping

Re: Ambari Mysql Database lost

rachida_el-hamm — Thu, 09 Sep 2021 13:41:12 GMT

Hello,

I'm facing the some issue, and I need help please.

I have 2 DB servers one Master and the second is slave, so after crash of the DB on the master node because the FS of the mysql is full, I try to establish the situation but I have the new problem after restarting the DB master correctly, all requests to tables give me the error ERROR 1932 (42S02): Table '*********' doesn't exist in engine so I suppose the DB is corrupt, so my problem is about database ambari only this DB not replicate on slave node and I have only the old baskup for Ambari,

my question is if I restore the DB ambari and I start Ambari with, can'I be sure to find the cluster on the some situation before the crash and be sure of all services restart correctly without impact or I have to execute this solution.

I'm not confident for reset/reinstall ambari server and agents, especially how to registre all hosts with services without impact.

Thank you for help.

Best Regards,

Re: Ambari Mysql Database lost

Shelton — Thu, 09 Sep 2021 14:19:11 GMT

@rachida_el-hamm

Here is a very good resource, sit back and sip your coffee or tea. It should help you resolve your MySQL issue

Happy hadooping

Re: Ambari Mysql Database lost

VidyaSargur — Tue, 14 Sep 2021 06:23:59 GMT

@rachida_el-hamm, did @Shelton's response help you resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.