Cloudera Manager Database Wiped

[#1]

I updated my Cloudera Manager Node from Ubuntu 12 to Ubuntu 14. during this upgrade I had to remove and re-install cloudera manager. This seems like it wiped my database. Is there any way that I can get it back?

Thank You

Jamie

[#2]

Jamie,

Two questions:

(1)

How did you remove Cloudera Manager?  (What commands, etc.)

(2)

How did you determine your database wiped?

[#3]

After upgrading the node I did these commands...

sudo apt-get remove cloudera-manager-agent cloudera-manager-daemons

Then re-installed the server, database and agent.

Cloudera Manager now opens with aclean slate. No data whatsoever...

[#4]

The first thing you need to do is recall what database you were using. 

How did you add Cloudera Manager back?  You may have accidentally installed the Embedded database (cloudera-manager-server-db-2).  I would run "rpm -qa |grep cloudera" to see.

Cloudera Manager reads the database connection information from /etc/cloudera-scm-server/db.properties.  So, if you know your database connection information, you should be able to find it there.

Please run ls -lrt /etc/cloudera-scm-server and provide the listing here so we can see if there were any saved db.properties files that may be there.   If so, one may contain your old settings.

If you find them or edit the db.properties, restart Cloudera Manager after verifying the db.properties file is correct.

[#5]

I installed both the sever and the server-db and started them. The server-db ran the init script and created a new db at this time. I have a backup of the db the way it was 6 months ago, but I did an CDH upgrade after that point.

This is our dev cluster, which was initially installed with the embedded db and it is again using that, but an empty one.  

[#6]

Uninstalling/reinstalling the embedded database should not remove or overwrite any data as far as I know.

You are saying that /var/lib/cloudera-scm-server-db has all new files dated from the day you reinstalled?

I'd check to see what is in /etc/cloudera-scm-server as I mentioned.  I'm wondering if you may have been using a different db/directory somehow...

If not, there really isn't much else to do but rebuild your Cloudera Manager configuration from scratch or try using the backup you have.

The backup will likely work OK.  When Cloudera Manager starts (using that old db) it will upgrade it as necessary.

Ben

[#7]

Hi Ben, 

It looks like the files in /var/lib/cloudera-scm-server-db/ are all new

ls -al /var/lib/cloudera-scm-server-db/data
total 116
drwx------ 15 cloudera-scm cloudera-scm 4096 Dec 7 11:26 .
drwxr-xr-x 3 cloudera-scm cloudera-scm 4096 Dec 7 10:47 ..
drwx------ 10 cloudera-scm cloudera-scm 4096 Dec 7 10:53 base
-rw------- 1 cloudera-scm cloudera-scm 264 Dec 7 10:53 generated_password.txt
drwx------ 2 cloudera-scm cloudera-scm 4096 Dec 7 11:27 global
drwx------ 2 cloudera-scm cloudera-scm 4096 Dec 7 10:53 pg_clog
-rw------- 1 cloudera-scm cloudera-scm 4581 Dec 7 10:53 pg_hba.conf
-rw------- 1 cloudera-scm cloudera-scm 1636 Dec 7 10:53 pg_ident.conf
drwx------ 4 cloudera-scm cloudera-scm 4096 Dec 7 10:53 pg_multixact
drwx------ 2 cloudera-scm cloudera-scm 4096 Dec 7 11:26 pg_notify
drwx------ 2 cloudera-scm cloudera-scm 4096 Dec 7 10:53 pg_serial
drwx------ 2 cloudera-scm cloudera-scm 4096 Dec 7 10:53 pg_snapshots
drwx------ 2 cloudera-scm cloudera-scm 4096 Dec 7 11:26 pg_stat
drwx------ 2 cloudera-scm cloudera-scm 4096 Dec 8 11:39 pg_stat_tmp
drwx------ 2 cloudera-scm cloudera-scm 4096 Dec 7 10:53 pg_subtrans
drwx------ 2 cloudera-scm cloudera-scm 4096 Dec 7 10:53 pg_tblspc
drwx------ 2 cloudera-scm cloudera-scm 4096 Dec 7 10:53 pg_twophase
-rw------- 1 cloudera-scm cloudera-scm 4 Dec 7 10:53 PG_VERSION
drwx------ 3 cloudera-scm cloudera-scm 4096 Dec 8 04:57 pg_xlog
-rw------- 1 cloudera-scm cloudera-scm 20430 Dec 7 10:53 postgresql.conf
-rw------- 1 cloudera-scm cloudera-scm 120 Dec 7 11:26 postmaster.opts
-rw------- 1 cloudera-scm cloudera-scm 112 Dec 7 11:26 postmaster.pid
-rw-rw-r-- 1 cloudera-scm cloudera-scm 24 Dec 7 10:53 scm.db.list
-rw-rw-r-- 1 cloudera-scm cloudera-scm 4 Dec 7 10:53 scm.db.list.20161207-105311

There are files inside /etc/cloudera-scm-sever 

ls -al /etc/cloudera-scm-server/
total 80
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Dec 7 11:07 .
drwxr-xr-x 171 root root 12288 Dec 7 10:48 ..
-rw------- 1 cloudera-scm cloudera-scm 6654 Dec 7 10:53 db.mgmt.properties
-rw------- 1 cloudera-scm cloudera-scm 1533 Nov 25 2013 db.mgmt.properties.20131125-101825
-rw------- 1 cloudera-scm cloudera-scm 2796 Nov 25 2013 db.mgmt.properties.20131125-102301
-rw------- 1 cloudera-scm cloudera-scm 4059 Nov 25 2013 db.mgmt.properties.20131125-115209
-rw------- 1 cloudera-scm cloudera-scm 5322 Dec 9 2014 db.mgmt.properties.20141209-145350
-rw------- 1 cloudera-scm cloudera-scm 5609 Dec 7 10:53 db.mgmt.properties.20161207-105311
-rw------- 1 cloudera-scm cloudera-scm 289 Dec 7 10:53 db.properties
-rw-r--r-- 1 cloudera-scm cloudera-scm 507 Nov 25 2013 db.properties.20131125-094853
-rw-r--r-- 1 cloudera-scm cloudera-scm 289 Nov 25 2013 db.properties.20131125-101825
-rw-r--r-- 1 cloudera-scm cloudera-scm 289 Nov 25 2013 db.properties.20131125-102301
-rw-r--r-- 1 cloudera-scm cloudera-scm 289 Nov 25 2013 db.properties.20131125-115209
-rw-r--r-- 1 cloudera-scm cloudera-scm 515 Dec 7 08:39 db.properties-old
-rw-r--r-- 1 cloudera-scm cloudera-scm 1521 Sep 16 14:56 log4j.properties

These files all contain different passwords in them. 

Jamie

[#8]

Hi Ben,

Do you think I could rebuild the database without losing data that is on the nodes? This is a kerberized cluster and I am afraid I will lose data if I keep messing with the db?

Thank You

Jamie

[#9 · Accepted solution]

You should be fine.  By design, Cloudera Manager does not remove any data from CDH.  To rebuild, you would basically add the services that you had before and choose the same locations for data that you had previously.  As mentioned, you could certainly use the 6-month-old database, too... CM will upgrade it the first time it starts.

Either way, your HDFS will not have been touched by the process of reinstalling Cloudera Manager and readding the services.

You will need to regenerate credentials after configuring Kerberos as the keytabs are stored in Cloudera Manager's database.  That will also not impact data, but it is another task you will need to perform.

Ben