Support Questions

Find answers, ask questions, and share your expertise

Data recovery in Erasure coding.

avatar
New Contributor

If a have 2 original block and 1 Parity block in HDFS according to the EC policy, then If I lose 2 of this 3 blocks, How HDFS is going to Recover the missing data. 

Block A  xor  Block B = Parity Block

If I lose any one of the block, I can see HDFS will be able to create the missing block. But if I lose any 2 block, I don't see how we will recover the missing 2 block  out of 1 block. 

If help me understand or redirect me to some helpful resources. 

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Hi @SalmanM , The fault tolerance in EC depends on no of parity blocks the chosen EC policy supports. For ex: take RS-3-2 policy which has got max 3 data blocks and 2 parity blocks. That means you can recover data by losing maximum of 2 blocks (inclusive of   parity or data blocks). You can refer the design guide here https://issues.apache.org/jira/browse/HDFS-7285 for more details or come back here for further info.

View solution in original post

3 REPLIES 3

avatar
Community Manager

@SalmanM, Welcome to our community! To help you get the best possible answer, I have tagged our HDFS experts @Asok who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Expert Contributor

Hi @SalmanM , The fault tolerance in EC depends on no of parity blocks the chosen EC policy supports. For ex: take RS-3-2 policy which has got max 3 data blocks and 2 parity blocks. That means you can recover data by losing maximum of 2 blocks (inclusive of   parity or data blocks). You can refer the design guide here https://issues.apache.org/jira/browse/HDFS-7285 for more details or come back here for further info.

avatar
New Contributor

Nice post and please provide more information. Thanks for sharing.