Created on 11-09-2023 11:47 PM - edited 11-10-2023 12:03 AM
If a have 2 original block and 1 Parity block in HDFS according to the EC policy, then If I lose 2 of this 3 blocks, How HDFS is going to Recover the missing data.
Block A xor Block B = Parity Block
If I lose any one of the block, I can see HDFS will be able to create the missing block. But if I lose any 2 block, I don't see how we will recover the missing 2 block out of 1 block.
If help me understand or redirect me to some helpful resources.
Created 11-23-2023 05:58 AM
Hi @SalmanM , The fault tolerance in EC depends on no of parity blocks the chosen EC policy supports. For ex: take RS-3-2 policy which has got max 3 data blocks and 2 parity blocks. That means you can recover data by losing maximum of 2 blocks (inclusive of parity or data blocks). You can refer the design guide here https://issues.apache.org/jira/browse/HDFS-7285 for more details or come back here for further info.
Created 11-15-2023 01:46 AM
@SalmanM, Welcome to our community! To help you get the best possible answer, I have tagged our HDFS experts @Asok who may be able to assist you further.
Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.
Regards,
Vidya Sargur,Created 11-23-2023 05:58 AM
Hi @SalmanM , The fault tolerance in EC depends on no of parity blocks the chosen EC policy supports. For ex: take RS-3-2 policy which has got max 3 data blocks and 2 parity blocks. That means you can recover data by losing maximum of 2 blocks (inclusive of parity or data blocks). You can refer the design guide here https://issues.apache.org/jira/browse/HDFS-7285 for more details or come back here for further info.
Created 11-23-2023 08:23 AM
Nice post and please provide more information. Thanks for sharing.