Data recovery in Erasure coding.

Question

If a have 2 original block and 1 Parity block in HDFS according to the EC policy, then If I lose 2 of this 3 blocks, How HDFS is going to Recover the missing data.

Block A xor Block B = Parity Block

If I lose any one of the block, I can see HDFS will be able to create the missing block. But if I lose any 2 block, I don't see how we will recover the missing 2 block out of 1 block.

If help me understand or redirect me to some helpful resources.

Majeti · Accepted Answer

Hi @SalmanM , The fault tolerance in EC depends on no of parity blocks the chosen EC policy supports. For ex: take RS-3-2 policy which has got max 3 data blocks and 2 parity blocks. That means you can recover data by losing maximum of 2 blocks (inclusive of   parity or data blocks). You can refer the design guide here https://issues.apache.org/jira/browse/HDFS-7285 for more details or come back here for further info.

VidyaSargur · Answer

@SalmanM, Welcome to our community! To help you get the best possible answer, I have tagged our HDFS experts @Asok who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.