Created 02-23-2018 04:38 PM
Hello there,
We are facing a big problem in our company :
Most of the /var directory was accidentally dropped after a bad log clean in Ambari-Server host.
In the removed files : .map files used to map content in PGSQL. The consequence is that the Ambari-Server can't start anymore. The databases are here but we get "could not find relation mapping" error, so we can't acces it well.
The problem : No DB Backup. No Snaphots. Overall, no directory backup. (Yes i know, this is very bad for a Production cluster...)
The Ambari Cluster is just over since yesterday.
I saw that thread today, but i'm not sure it will be the best issue for us :
https://community.hortonworks.com/questions/6703/ambari-server-installation-after-cluster-setup.html
Infos :
HDP 2.6.1 / Ambari 2.5.1 / PGSQL 9.5
1 Gateway with Ambari-Server (the impacted server) / 2 Masters / 4 Slaves
Our goal is to save the data on the Datanodes and get an Ambari Cluster back.
Any ideas or similar past experiences ?
Thanks all
EDIT 1 : Impacted databases on the host are Ambari / Hive / Oozie / Hue
Created 02-24-2018 07:02 PM
That's just a warning that you have existing processes on the machines. If you let it uninstall packages or delete user accounts, you'll have downtime on the cluster, and services might not stop gracefully, so you risk additional corruption.
I've added machines like this that are provisioned by Puppet, and so there are some extra background services running, but I just ignore that warning, and Ambari has set them up fine.
Regarding the Hive Metastore, if you have set it up to use an external Postgres/MySQL database (recommended), I would probably first let Ambari first install the embedded Derby database for Hive, then manually edit the hive-site XML to point to the old one.
Created 03-02-2018 03:53 PM
You are right and we are are currently in Ambari configuration. As the Ambari database was dropped, there is nothing left when you add the hosts.
But we can use XML files on the host to find most of the parameters, they have not been deleted.
We re-installed first, now we are looking for our past specific parameters. This is why backup is important !