Reply
Explorer
Posts: 14
Registered: ‎08-14-2015

Data integrity check in HDFS..

Hi,

 

We usually check if .iso  download is fine or not using provided checksum information.

 

Do we have any similar mechanism in HDFS? How to check if the file transfer is complete and successful in HDFS?

I tried using curl command, but it not giving md5 checksum.

 

 

In the Hadoop file system check sum value  is md5 of md5of 512crc32 algorithm  and local file system using md5 algorithm . so comparison is failing . And also ,I tried  to implementation md5 of md5of 512crc32 algorithm  for local file system that algorithm is not supporting local file system. It allowed only for distributed systems.

 

Could you please help solution  for comparing checksum for a file present initially in local file system and after it is transferred to HDFS.

 

Thanks a lot.

 

Announcements