Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Running Hbase backup command from remote machine fails

Highlighted

Running Hbase backup command from remote machine fails

New Contributor

Hi ,

Currently I am writing a python script which is in my local machine and I am calling a shell script which is residing in remote server. The Shell script has the Hbase Backup command to push backup to S3. Shell script works fine taking backup and pushing to s3 when executed in that remote machine. But when I call that script from my local machine through python it errors out.

Below is the command which I am running in my shell script.

 hbase backup create full s3a://$AccessKey:$SecretKey@$BucketPath -set $BackupSet

I get this below error

ERROR [main] util.AbstractHBaseTool: Error running command-line tool
java.io.IOException: Failed of exporting snapshot snapshot_1521106075558_default_tablename to s3a://AccessKey:SecretKey@swe-backup-test/backup_1521106069410/default/tablename/ with reason code 1
\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
\tat java.lang.reflect.Constructor.newInstance(Constructor.java:526)
\tat org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
\tat org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
\tat org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
\tat org.apache.hadoop.hbase.client.HBaseAdmin$TableBackupFuture.convertResult(HBaseAdmin.java:2787)
\tat org.apache.hadoop.hbase.client.HBaseAdmin$TableBackupFuture.convertResult(HBaseAdmin.java:2766)
\tat org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4769)
\tat org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4703)
\tat org.apache.hadoop.hbase.client.HBaseAdmin.get(HBaseAdmin.java:2744)
\tat org.apache.hadoop.hbase.client.HBaseAdmin.backupTables(HBaseAdmin.java:2760)
\tat org.apache.hadoop.hbase.client.HBaseBackupAdmin.backupTables(HBaseBackupAdmin.java:243)
\tat org.apache.hadoop.hbase.backup.impl.BackupCommands$CreateCommand.execute(BackupCommands.java:197)
\tat org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:111)
\tat org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:126)
\tat org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
\tat org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
\tat org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:131)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Failed of exporting snapshot snapshot_1521106075558_default_US_tablename to s3a://Accesskey:SecretKey@swe-backup-test/backup_1521106069410/default/tablename/ with reason code 1
\tat org.apache.hadoop.hbase.backup.master.FullTableBackupProcedure.snapshotCopy(FullTableBackupProcedure.java:321)
\tat org.apache.hadoop.hbase.backup.master.FullTableBackupProcedure.executeFromState(FullTableBackupProcedure.java:577)
\tat org.apache.hadoop.hbase.backup.master.FullTableBackupProcedure.executeFromState(FullTableBackupProcedure.java:69)
\tat org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:107)
\tat org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:500)
\tat org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1086)
\tat org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:888)
\tat org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:841)
\tat org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$200(ProcedureExecutor.java:77)
\tat org.apache.hadoop.hbase.procedure2.ProcedureExecutor$1.run(ProcedureExecutor.java:443)

After some research I found the below link which has the same error which I am receiving when I call my python from my local machine.

https://issues.apache.org/jira/browse/HBASE-15853

In that link , they say /user/hbase directory should be present in hdfs and should be writable. I already have that directory present in my hdfs and also i have made it writable. But still facing the same error.

can anybody please help me out to resolve this issue. It is urgent. @Jay Kumar SenSharma

Thanks

3 REPLIES 3

Re: Running Hbase backup command from remote machine fails

Can you please post snapshot of your logs?

Re: Running Hbase backup command from remote machine fails

Expert Contributor

Hi @Swetha Nelwad,

I think you are using few parameters in shell script. Can you check whether parameters are translating into values?

At first, you can hardcode the values and simply call script to see whether it works.

hbase backup create full s3a://$AccessKey:$SecretKey@$BucketPath -set $BackupSet

-Shubham

Re: Running Hbase backup command from remote machine fails

Can I add that putting secrets in your s3a:// path is dangerous as it will end up in hadoop logs across the cluster

Best: put them in a JCEKs file in HDFS or other secure keystore

Good: have some options in the hadoop/hbase configurations

Weak: setting them on the command line with -D options (visible with a ps command)