I am in an uncomfortable situation. Some weeks ago I migrated a HDP 2.2 cluster to HDP 2.6 (using HDP 2.4 in the middle). I made a backup of the Oracle DB before each upgrade. I just haven't done one at the end. 2 day's ago a after a disk cash and a hard reboot of the Oracle instance, my Ambari database has a corrupt datafile. We also lost all the redologs ...
So I have a DB dump for HDP 2.2 (Ambari 2.0) and HDP 2.4 (Ambari 2.2) but no redologs.
The guys from Ooacle say I have to re-import the old dump, but in this case my cluster won't work anymore. (Since Ambari will push the old config to a newer HDP installation.)
Do I have to remove the whole cluster and reinstall, or is there another solution ?
it's not the Hadoop data I am worried about. It's how Ambari will handle the configuration.
Let's say I am using Ambari 2.5 with HDP 2.6
1) I need to restore a DB dump for Ambari 2.0
2) All the migration configuration will be lost (of ambari and the HDP services)
3) I guess I have to remove the whole cluster and reinstall it. (Without removing the data)
If you have backups, let me point you to a potential solution.
0) Stop your ambari-agents on all hosts > HDP keeps running but there is no try to join the crashed Ambari-Server.
1) Restore your Ambari backup from HDP 2.4 (Ambari 2.2) on your existing Ambari-Server > You will have all your parameters back, but the Ambari hosts won't point on the server.
2) Create a new Ambari-Server with HDP 2.6 (Ambari 2.5 ?), same version you had before the crash > This will be your new Ambari-Server.
Here is the point : You will have an old Ambari-Server with most of your parameters on it (model) and a new one to build.
Like that, you can rebuild your parameters step by step, on the new Ambari-Server, when your hosts are added.
3) Modify the ambari-agents Ambari-Server hostname to target the new server and restart them.
4) Add your host with exact past roles.
What do you think about it ?