question Re: Data recovery in Erasure coding. in Support Questions

Data recovery in Erasure coding.

SalmanM — Fri, 10 Nov 2023 08:03:57 GMT

If a have 2 original block and 1 Parity block in HDFS according to the EC policy, then If I lose 2 of this 3 blocks, How HDFS is going to Recover the missing data.

Block A xor Block B = Parity Block

If I lose any one of the block, I can see HDFS will be able to create the missing block. But if I lose any 2 block, I don't see how we will recover the missing 2 block out of 1 block.

If help me understand or redirect me to some helpful resources.

Re: Data recovery in Erasure coding.

VidyaSargur — Wed, 15 Nov 2023 09:46:59 GMT

@SalmanM, Welcome to our community! To help you get the best possible answer, I have tagged our HDFS experts @Asok who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Data recovery in Erasure coding.

Majeti — Thu, 23 Nov 2023 13:58:29 GMT

Hi @SalmanM , The fault tolerance in EC depends on no of parity blocks the chosen EC policy supports. For ex: take RS-3-2 policy which has got max 3 data blocks and 2 parity blocks. That means you can recover data by losing maximum of 2 blocks (inclusive of parity or data blocks). You can refer the design guide here https://issues.apache.org/jira/browse/HDFS-7285 for more details or come back here for further info.

Re: Data recovery in Erasure coding.

paxycrone — Thu, 23 Nov 2023 16:23:39 GMT

Nice post and please provide more information. Thanks for sharing.