how to recover hbase using hdfs data directory

[#1]

My old hdfs data directory location - /apps/hbase/data

My new hdfs data directory location - /apps/hbase/data2

Hbase table Name - CUTOFF2

create 'CUTOFF2', {NAME => '1'}

I am doing following steps to recover data. But not working. Please tell me where I am wrong-

hadoop fs -ls /apps/hbase/data/data/default/CUTOFF2/4c8d68c329cdb6d73d4094fd64e5e37d/1/d321dfcd3b1245d2b5cc2ec1aab3a9f2 hadoop fs -ls /apps/hbase/data2/data/default/CUTOFF2/8f1aff44991e1a08c6a6bbf9c2546cf6/1

put 'CUTOFF2' , 'samplerow', '1:1' , 'sampledata' count 'CUTOFF2'

su - hbase

hadoop fs -cp /apps/hbase/data/data/default/CUTOFF2/4c8d68c329cdb6d73d4094fd64e5e37d/1/d321dfcd3b1245d2b5cc2ec1aab3a9f2 /apps/hbase/data2/data/default/CUTOFF2/8f1aff44991e1a08c6a6bbf9c2546cf6/1

major_compact 'CUTOFF2'

Please correct my steps so recovery works.

[#2]

In hbase-site.xml, you need to change the "hbase.rootdir" property to your new location.

[#3 · Accepted solution]

Hi @Raja Ray, here are the steps for recover Hfiles in another hdfs directory:

1. Shutdown the hbase with old hdfs path.

2. Change 'hbase.rootdir' to new path and restart hbase.

3. Create table 'CUTOFF2', so that new htable structure will be created in new hdfs path, and of course, it's empty.

4. Use distcp to copy hfile(s) from old path to new path in case the hfile(s) are very huge.

5. Do a 'hbase hbck' on the new hbase, and there should be something wrong with the 'CUTOFF2'.

6. Do a 'hbase hbck -repair' on the problematic table and it will finalize the recovery.

7. Done

[#4]

Thanks Victor. I will follow your steps and will let you know.

[#5]

Hi @Victor Xu,

I followed your steps. It is working fine.

But i needed to restart hbase

Can you please suggest me any other way where I don't need to restart hbase service.

Thanks,

Raja

[#6]

Hi @Raja Ray,

1. Which version of hbase are you using?

2. When performing my steps, is there any specific error log that you can share with me?

3. Could you elaborate on your use case?

Thanks,

Victor

[#7]

Hi @Victor Xu,

I followed your steps. It is working fine.

But i needed to restart hbase

Can you please suggest me any other way where I don't need to restart hbase service.

Thanks,

Raja

[#8]

Ok, I understand. But even if you just want to change hdfs root directory for a running hbase cluster, you'll need a restart to make it work.

Do you mean you've already change the root path to '/apps/hbase/data2' before starting your current hbase cluster?

[#9]

In other words, there's no 'hot switch' for this 'hbase.rootdir' parameter. If you want to change it, you have to restart hbase to make it work.

[#10]

Hi @Raja Ray,

I checked but HBase rolling upgrade won't help here either, because HMaster and RS both use this 'hbase.rootdir' in the runtime and only changing part of them would cause data inconsistencies. So my suggestion would be create a smaller temporary hbase cluster to handle all the production requests and do a quick restart on the main hbase cluster. Modifying 'hbase.rootdir' really needs downtime.

Hope that will help.

Thanks,

Victor