URGENT: Distcp -update -delete is not deleting files

snm1523 — Fri, 16 Jun 2023 18:11:20 GMT

Hello,

We are on CDP 7.1.7 SP1 and as per my understanding when DistCp is executed with below parameters, it does the following:

1. update: this will update the files in destination that have got modified at source

2. delete: this will delete the files in destination that exist in destination but not in source.

Apart feom above, it will copy over any new files at source that do not exist in destination.

While copy and update is happening, it does not delete the files instead duplicates them and hence, the data count at destination is double than the source.

Kindly help urgently as we are at the verge of initiating heavy data copy.

Thanks

snm1523

Re: URGENT: Distcp -update -delete is not deleting files

Asfahan — Mon, 20 May 2024 06:36:33 GMT

Hello , can you please share the code of distcp you were using with all necessary parameter
ALso did you try to run the same in Debug console to understand more

HADOOP_ROOT_LOGGER=DEBUG,console

question Re: URGENT: Distcp -update -delete is not deleting files in Support Questions

URGENT: Distcp -update -delete is not deleting files

Re: URGENT: Distcp -update -delete is not deleting files