Hello @ashinde    I mean if we delete data on the source CDP for whatever reason (purge, archiving, dataset rebuild) how to capture those events and replicate the delete action on the target CDP.   it seems that all of my proposal will only able to add data on the target. ... View more

I have several ideas in mind :   1) Nifi process at the file system level : capture data with ListHDFS + FetchHDFS --> ingest data with PuHDFS But what about file delete ? I was thinking to use GetHDFSEvents to capture "unlink" events and replicate events with DeleteHDFS But it seems that GetHDFSEvents is not compatible with ADLS Gen2 storage   2) Distcp Again seems working with new data and updated file but I don't understand how it can handle deletes (except if we drop the target data before a full copy)   3) AzCopy Only compatible with ADLS (but I imagine that a similar tool is available for S3 buckets) with "azcopy sync" option https://docs.microsoft.com/fr-fr/azure/storage/common/storage-ref-azcopy-sync   4) Nifi process at Hive level Not sure if it's very elegant : capture data with SelectHive3QL (Avro output) Ingest data with PutHive3Streaming But not sure how to manage deletes   Any best practice or other better idea ? ... View more