Support Questions
Find answers, ask questions, and share your expertise

Ambari Mysql Database lost

Solved Go to solution

Ambari Mysql Database lost

New Contributor

Hello everyone

I have development cluster where I lost my ambari database which was on  a mysql database.

Is there anyway to reinstall ambari server and have my cluster back up again ?

 

Thank you for your help

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Ambari Mysql Database lost

Mentor

@moubaba 

I know its a dompting task but unfortunately, I could create a link by an article for Jordan Moore let me try to elaborate. 

 

Simple advice always run a crontab job to dump all your databases or at leat the ambari DB  nightly !

 

Firstly Ambari doesn't control any data residing on the data nodes, so you should be safe there. Stop all ambari-agents in the cluster, maybe even uninstalling it so as to let all the Hadoop components remain running "in the dark".

Install and set up a new Ambari server, add a cluster but do not register any  hosts [very important]. If you uninstalled the ambari -agents on the nodes please do re-install ensure the ambari-server and the agents are the same versions.

Do the below if you removed Ambari and the agents

# yum repolist

On the ambari-server

# yum install ambari-server

# yum install ambari-agent

On the other hosts

# yum install ambari-agent

Reconfigure manually the ambari agents [ambari-agent.ini] to point at the new Ambari server address, and start them.

 

Add the hosts in the Ambari server UI, selecting "manual registration" option the hosts register successfully since the agents are running a point to note install and configure an ambari-agent on the Ambari server to point to itself !!!!

After this, you get the option of installing clients and servers.

Now, you could try to "reinstall" what is already there, but you might want to deselect all the servers on the data node column.

In theory, it will try to perform OS package installation, and say that the service already exists, and doesn't error out. If it does error, then restart the install process, but deselect everything. At which point, it should continue and now you have Ambari back up and running with all the hosts monitored, just with no processes to configure.

To add the services back, you would need to use the Ambari REST API to add back in the respective Services, Components, and Host Components that you have running on the cluster. If you can't remember all the Services, Components, and Host Components then go to each host and do a process check to see what's running.

 

Variables

export AMBARI_USER=admin
export AMBARI_PASSWD=admin
export CLUSTER_NAME=<New_Cluster_name>
export AMBARI_SERVER=<Your_new_ambari_server_FQDN>
export AMBARI_PORT=8080


List all services related to hive

In the below example I am listing all hive components 

curl -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X GET "http://$AMBARI_SERVER:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/services/HIVE"


Adding back pig

curl -k -u $AMBARI_USER:$AMBARI_PASSWD -H "X-Requested-By:ambari" -i -X POST -d '{"RequestInfo":{"context":"Install PIG"}, "Body":{"HostRoles":{"state":"INSTALLED"}}}' 'http://'$AMBARI_SERVER':$AMBARI_PORT/api/v1/clusters/'$CLUSTER_NAME'/services/PIG/components/PIG'

This is the time to get your hands dirty This isn't production so it should train you for a real situation
Happy hadooping !!

View solution in original post

5 REPLIES 5

Re: Ambari Mysql Database lost

Mentor

@moubaba 

I know its a dompting task but unfortunately, I could create a link by an article for Jordan Moore let me try to elaborate. 

 

Simple advice always run a crontab job to dump all your databases or at leat the ambari DB  nightly !

 

Firstly Ambari doesn't control any data residing on the data nodes, so you should be safe there. Stop all ambari-agents in the cluster, maybe even uninstalling it so as to let all the Hadoop components remain running "in the dark".

Install and set up a new Ambari server, add a cluster but do not register any  hosts [very important]. If you uninstalled the ambari -agents on the nodes please do re-install ensure the ambari-server and the agents are the same versions.

Do the below if you removed Ambari and the agents

# yum repolist

On the ambari-server

# yum install ambari-server

# yum install ambari-agent

On the other hosts

# yum install ambari-agent

Reconfigure manually the ambari agents [ambari-agent.ini] to point at the new Ambari server address, and start them.

 

Add the hosts in the Ambari server UI, selecting "manual registration" option the hosts register successfully since the agents are running a point to note install and configure an ambari-agent on the Ambari server to point to itself !!!!

After this, you get the option of installing clients and servers.

Now, you could try to "reinstall" what is already there, but you might want to deselect all the servers on the data node column.

In theory, it will try to perform OS package installation, and say that the service already exists, and doesn't error out. If it does error, then restart the install process, but deselect everything. At which point, it should continue and now you have Ambari back up and running with all the hosts monitored, just with no processes to configure.

To add the services back, you would need to use the Ambari REST API to add back in the respective Services, Components, and Host Components that you have running on the cluster. If you can't remember all the Services, Components, and Host Components then go to each host and do a process check to see what's running.

 

Variables

export AMBARI_USER=admin
export AMBARI_PASSWD=admin
export CLUSTER_NAME=<New_Cluster_name>
export AMBARI_SERVER=<Your_new_ambari_server_FQDN>
export AMBARI_PORT=8080


List all services related to hive

In the below example I am listing all hive components 

curl -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X GET "http://$AMBARI_SERVER:$AMBARI_PORT/api/v1/clusters/$CLUSTER_NAME/services/HIVE"


Adding back pig

curl -k -u $AMBARI_USER:$AMBARI_PASSWD -H "X-Requested-By:ambari" -i -X POST -d '{"RequestInfo":{"context":"Install PIG"}, "Body":{"HostRoles":{"state":"INSTALLED"}}}' 'http://'$AMBARI_SERVER':$AMBARI_PORT/api/v1/clusters/'$CLUSTER_NAME'/services/PIG/components/PIG'

This is the time to get your hands dirty This isn't production so it should train you for a real situation
Happy hadooping !!

View solution in original post

Re: Ambari Mysql Database lost

New Contributor

Thanks Shelton for your  very detailled response. It is really helpful.

Concerning the adding back services with the API : does it install the services again ?

For example, if I add back Hive services does it install hive again ?

Re: Ambari Mysql Database lost

New Contributor

I got another question also.

What about the views ? Like File View and Hive View, can I create them back ?

 

Re: Ambari Mysql Database lost

New Contributor

Hello Shelton

Any help to recreate Ambari Views ?

Re: Ambari Mysql Database lost

Mentor

@moubaba 

Here is a fantastic document by Artem Ervits  Ambari Views REST API Overview

Hope that answers your query 

 

Happy hadooping