Ambari Mysql Database lost

[#1]

Hello everyone

I have development cluster where I lost my ambari database which was on  a mysql database.

Is there anyway to reinstall ambari server and have my cluster back up again ?

Thank you for your help

[#2 · Accepted solution]

@moubaba 

I know its a dompting task but unfortunately, I could create a link by an article for Jordan Moore let me try to elaborate. 

Simple advice always run a crontab job to dump all your databases or at leat the ambari DB  nightly !

Firstly Ambari doesn't control any data residing on the data nodes, so you should be safe there. Stop all ambari-agents in the cluster, maybe even uninstalling it so as to let all the Hadoop components remain running "in the dark".

Install and set up a new Ambari server, add a cluster but do not register any  hosts [very important]. If you uninstalled the ambari -agents on the nodes please do re-install ensure the ambari-server and the agents are the same versions.

Do the below if you removed Ambari and the agents

# yum repolist

On the ambari-server

# yum install ambari-server

# yum install ambari-agent

On the other hosts

# yum install ambari-agent

Reconfigure manually the ambari agents [ambari-agent.ini] to point at the new Ambari server address, and start them.

Add the hosts in the Ambari server UI, selecting "manual registration" option the hosts register successfully since the agents are running a point to note install and configure an ambari-agent on the Ambari server to point to itself !!!!

After this, you get the option of installing clients and servers.

Now, you could try to "reinstall" what is already there, but you might want to deselect all the servers on the data node column.

In theory, it will try to perform OS package installation, and say that the service already exists, and doesn't error out. If it does error, then restart the install process, but deselect everything. At which point, it should continue and now you have Ambari back up and running with all the hosts monitored, just with no processes to configure.

To add the services back, you would need to use the Ambari REST API to add back in the respective Services, Components, and Host Components that you have running on the cluster. If you can't remember all the Services, Components, and Host Components then go to each host and do a process check to see what's running.

Variables

export AMBARI_USER=admin
export AMBARI_PASSWD=admin
export CLUSTER_NAME=<New_Cluster_name>
export AMBARI_SERVER=<Your_new_ambari_server_FQDN>
export AMBARI_PORT=8080

List all services related to hive

In the below example I am listing all hive components 

curl -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X GET "http://$AMBARI_SERVER:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/services/HIVE"

Adding back pig

curl -k -u $AMBARI_USER:$AMBARI_PASSWD -H "X-Requested-By:ambari" -i -X POST -d '{"RequestInfo":{"context":"Install PIG"}, "Body":{"HostRoles":{"state":"INSTALLED"}}}' 'http://'$AMBARI_SERVER':$AMBARI_PORT/api/v1/clusters/'$CLUSTER_NAME'/services/PIG/components/PIG'

This is the time to get your hands dirty 🙂 This isn't production so it should train you for a real situation
Happy hadooping !!

[#3]

Thanks Shelton for your  very detailled response. It is really helpful.

Concerning the adding back services with the API : does it install the services again ?

For example, if I add back Hive services does it install hive again ?

[#4]

I got another question also.

What about the views ? Like File View and Hive View, can I create them back ?

[#5]

Hello Shelton

Any help to recreate Ambari Views ?

[#6]

@moubaba 

Here is a fantastic document by Artem Ervits  Ambari Views REST API Overview

Hope that answers your query 

Happy hadooping

[#7]

Hello,

I'm facing the some issue, and I need help please.

I have 2 DB servers one Master and the second is slave, so after crash of the DB on the master node because the FS of the mysql is full, I try to establish the situation but I have the new problem after restarting the DB master correctly, all requests to tables give me the error ERROR 1932 (42S02): Table '*********' doesn't exist in engine so I suppose the DB is corrupt, so my problem is about database ambari only this DB not replicate on slave node and I have only the old baskup for Ambari, 

my question is if I restore the DB ambari and I start Ambari with, can'I be sure to find the cluster on the some situation before the crash and be sure of all services restart correctly without impact or I have to execute  this  solution.

I'm not confident for reset/reinstall ambari server and agents, especially how to registre all hosts with services without impact.

Thank you for help.

Best Regards,

[#8]

@rachida_el-hamm 

Here is a very good resource, sit back and sip your coffee or tea. It should help you resolve your MySQL issue 

Happy hadooping

[#9]

@rachida_el-hamm, did @Shelton's response help you resolve your issue?  If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.