Support Questions
Find answers, ask questions, and share your expertise

ambari database corrupt

ambari database corrupt

Explorer

Hi,

I am in an uncomfortable situation. Some weeks ago I migrated a HDP 2.2 cluster to HDP 2.6 (using HDP 2.4 in the middle). I made a backup of the Oracle DB before each upgrade. I just haven't done one at the end. 2 day's ago a after a disk cash and a hard reboot of the Oracle instance, my Ambari database has a corrupt datafile. We also lost all the redologs ...

So I have a DB dump for HDP 2.2 (Ambari 2.0) and HDP 2.4 (Ambari 2.2) but no redologs.

The guys from Ooacle say I have to re-import the old dump, but in this case my cluster won't work anymore. (Since Ambari will push the old config to a newer HDP installation.)

Do I have to remove the whole cluster and reinstall, or is there another solution ?

Thanks

Manfred

3 REPLIES 3

Re: ambari database corrupt

Mentor

@Manfred PAUL

Here is a response by @Jordan Moore which addresses that issue, read carefully and adopt it to your Oracle DB installation but that should work. As reiterated "Ambari doesn't control any data residing on the datanodes, so you should be safe there"

Good luck

Re: ambari database corrupt

Explorer

@Geoffrey Shelton Okot

Hi,

it's not the Hadoop data I am worried about. It's how Ambari will handle the configuration.

Let's say I am using Ambari 2.5 with HDP 2.6

1) I need to restore a DB dump for Ambari 2.0

2) All the migration configuration will be lost (of ambari and the HDP services)

3) I guess I have to remove the whole cluster and reinstall it. (Without removing the data)

Manfred

Re: ambari database corrupt

@Manfred PAUL

If you have backups, let me point you to a potential solution.

0) Stop your ambari-agents on all hosts > HDP keeps running but there is no try to join the crashed Ambari-Server.

1) Restore your Ambari backup from HDP 2.4 (Ambari 2.2) on your existing Ambari-Server > You will have all your parameters back, but the Ambari hosts won't point on the server.

2) Create a new Ambari-Server with HDP 2.6 (Ambari 2.5 ?), same version you had before the crash > This will be your new Ambari-Server.

Here is the point : You will have an old Ambari-Server with most of your parameters on it (model) and a new one to build.

Like that, you can rebuild your parameters step by step, on the new Ambari-Server, when your hosts are added.

3) Modify the ambari-agents Ambari-Server hostname to target the new server and restart them.

4) Add your host with exact past roles.

What do you think about it ?