question Re: how can i copy hbase data from replication set cluster to ec cluster in Support Questions

how can i copy hbase data from replication set cluster to ec cluster

Eren — Thu, 20 Apr 2023 09:58:41 GMT

hello guys, I need to copy hbase data from replication set cluster to ec cluster, i have tried hbase snapshot export and hadoop distcp ,they look like throw the same error: Not replicated yet, i think this was caused by when they transporting ,they will check if destination cluster replication num was same as local, but ignore if destination cluster used replication set. how can i do it ,thx.

There is all the errors.

snapshot export:

2023-04-19 19:44:10,199 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK* blk_-9223372036854751104_9943 is COMMITTED but not COMPLETE(numNodes= 5 >= minimum = 3) in file /hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea508
2023-04-19 19:44:10,200 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call Call#31 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 172.31.0.146:52210
org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet: /hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea508
at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:181)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2661)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:872)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:550)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)

hadoop distcp:

java.io.IOException: Unable to close file because the last block does not have enough number of replicas.
at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2521)
at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2482)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2447)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:258)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:183)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:123)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)

Re: how can i copy hbase data from replication set cluster to ec cluster

DianaTorres — Thu, 20 Apr 2023 15:03:12 GMT

@Eren Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our HBase experts @blizano and @pajoshi who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: how can i copy hbase data from replication set cluster to ec cluster

blizano — Thu, 20 Apr 2023 19:34:59 GMT

Hello @Eren

Mabe this kb can be useful https://community.cloudera.com/t5/Customer/HDFS-write-occasionally-fails-with-error-message-quot-Unable/ta-p/81420

The block chain replication for HDFS does not seem to be able to complete:

snapshot export:

Then make sure that the destination cluster has enough number of datanodes available and with space based on the replication factor or the hbase file, in this case

/hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea508

You can reconfirm the RF of the file by running any of the following commands:

hdfs fsck

/hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea508

hdfs dfs -ls

/hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea508.

In the last one, the second column will tell the RF applied to the file.

You can also check in the destination cluster if the datanodes are distributed in racks, if so, check that each rack has enough number of datanodes available and with space.

Hope this helps.

Re: how can i copy hbase data from replication set cluster to ec cluster

Eren — Fri, 21 Apr 2023 02:37:38 GMT

hi blizano , thanks for reply, im sure this issue is caused by ec Policy, if i unset ec on /hbase, hbase snapshot export works. and i have tried set dfs.client.block.write.locateFollowingBlock.retries to 10 , but it's not useful, that throw the same error as before.

and i run fsck, it shows:

/hbase/.hbase-snapshot/test_snapshot/.snapshotinfo: Under replicated BP-870847850-192.168.30.174-1680829859143:blk_-9223372036854767824_8445. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)

/hbase/.hbase-snapshot/test_snapshot/data.manifest: Under replicated BP-870847850-192.168.30.174-1680829859143:blk_-9223372036854767808_8446. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

in fact my Target Replicas is not 3 , because i have set ec on /hbase . and i tried set dfs.replication to 1 ,it not useful too.

i run du on /hbase/archive/data/default/test/ ,it shows 1.1 M 384 M, this is strange, original file is only 1.1M,but total size is 384 M.

Re: how can i copy hbase data from replication set cluster to ec cluster

Eren — Fri, 21 Apr 2023 10:00:50 GMT

i fix this issue by run command on destination cluster,i think it caused by original version is too old to support ec (hadoop-2.7.5)