The HDFS replication is failing even after the successful completion of the MapReduce job with the following error:
ERROR distcp.DistCp: Job failed to copy 709 files/dirs. Please check Copy Status.csv file or Error Status.csv file for error messages
INFO distcp.DistCp: Used diff: false
INFO distcp.DistCp: Distcp failed to copy some files/dirs, skipping delete and rename snapshot operations on Source
INFO distcp.DistCp: Distcp failed to copy some files/dirs, skipping delete and rename snapshot operations on Target
The cluster is running with CDH 5.15.1. There are enough resources available in the destination cluster for the running of MapReduce job for DistCp. This is critical. Please assist.
Hi @urbanlad20 ,
Could you please share the Copy Status.csv file and/or Error Status.csv file so we can find out more details of what happened?
There is nothing shown in the Error status file and its kind of a dead end. Also, KMS has been enabled in the Source cluster. Tried manually doing a DistCp -update which was not working initially, but after removing the HDFS user from the blacklist in the kms-acls.xml file, it worked out fine.
Attached is the snap of the Error-status.csv file