Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to recover hbase using hdfs data directory

avatar
Expert Contributor

My old hdfs data directory location - /apps/hbase/data

My new hdfs data directory location - /apps/hbase/data2

Hbase table Name - CUTOFF2

create 'CUTOFF2', {NAME => '1'}

I am doing following steps to recover data. But not working. Please tell me where I am wrong-

hadoop fs -ls /apps/hbase/data/data/default/CUTOFF2/4c8d68c329cdb6d73d4094fd64e5e37d/1/d321dfcd3b1245d2b5cc2ec1aab3a9f2 hadoop fs -ls /apps/hbase/data2/data/default/CUTOFF2/8f1aff44991e1a08c6a6bbf9c2546cf6/1

put 'CUTOFF2' , 'samplerow', '1:1' , 'sampledata' count 'CUTOFF2'

su - hbase

hadoop fs -cp /apps/hbase/data/data/default/CUTOFF2/4c8d68c329cdb6d73d4094fd64e5e37d/1/d321dfcd3b1245d2b5cc2ec1aab3a9f2 /apps/hbase/data2/data/default/CUTOFF2/8f1aff44991e1a08c6a6bbf9c2546cf6/1

major_compact 'CUTOFF2'

Please correct my steps so recovery works.

1 ACCEPTED SOLUTION

avatar
Rising Star

Hi @Raja Ray, here are the steps for recover Hfiles in another hdfs directory:

1. Shutdown the hbase with old hdfs path.

2. Change 'hbase.rootdir' to new path and restart hbase.

3. Create table 'CUTOFF2', so that new htable structure will be created in new hdfs path, and of course, it's empty.

4. Use distcp to copy hfile(s) from old path to new path in case the hfile(s) are very huge.

5. Do a 'hbase hbck' on the new hbase, and there should be something wrong with the 'CUTOFF2'.

6. Do a 'hbase hbck -repair' on the problematic table and it will finalize the recovery.

7. Done

View solution in original post

12 REPLIES 12

avatar
Contributor

In hbase-site.xml, you need to change the "hbase.rootdir" property to your new location.

avatar
Rising Star

Hi @Raja Ray, here are the steps for recover Hfiles in another hdfs directory:

1. Shutdown the hbase with old hdfs path.

2. Change 'hbase.rootdir' to new path and restart hbase.

3. Create table 'CUTOFF2', so that new htable structure will be created in new hdfs path, and of course, it's empty.

4. Use distcp to copy hfile(s) from old path to new path in case the hfile(s) are very huge.

5. Do a 'hbase hbck' on the new hbase, and there should be something wrong with the 'CUTOFF2'.

6. Do a 'hbase hbck -repair' on the problematic table and it will finalize the recovery.

7. Done

avatar
Expert Contributor

Thanks Victor. I will follow your steps and will let you know.

avatar
Expert Contributor

Hi @Victor Xu,

I followed your steps. It is working fine.

But i needed to restart hbase

Can you please suggest me any other way where I don't need to restart hbase service.

Thanks,

Raja

avatar
Rising Star

Hi @Raja Ray,

1. Which version of hbase are you using?

2. When performing my steps, is there any specific error log that you can share with me?

3. Could you elaborate on your use case?

Thanks,

Victor

avatar
Expert Contributor

Hi @Victor Xu,

I followed your steps. It is working fine.

But i needed to restart hbase

Can you please suggest me any other way where I don't need to restart hbase service.

Thanks,

Raja

avatar
Rising Star

Ok, I understand. But even if you just want to change hdfs root directory for a running hbase cluster, you'll need a restart to make it work.

Do you mean you've already change the root path to '/apps/hbase/data2' before starting your current hbase cluster?

avatar
Rising Star

In other words, there's no 'hot switch' for this 'hbase.rootdir' parameter. If you want to change it, you have to restart hbase to make it work.

avatar
Rising Star

Hi @Raja Ray,

I checked but HBase rolling upgrade won't help here either, because HMaster and RS both use this 'hbase.rootdir' in the runtime and only changing part of them would cause data inconsistencies. So my suggestion would be create a smaller temporary hbase cluster to handle all the production requests and do a quick restart on the main hbase cluster. Modifying 'hbase.rootdir' really needs downtime.

Hope that will help.

Thanks,

Victor