What limitations are we talking about here? Sorry, I don't understand your question. If you are asking about DIstCP options supported in HDFS Mirroirng, currently below options are supported maxMaps mapBandwidth Below additional options can be supported by using workaround given below: overwrite ignoreErrors skipChecksum removeDeletedFiles preserveBlockSize preserveReplicationNumber preservePermission Please modify the WF hdfs-replication-workflow.xml as below. After distcpMapBandwidth add below content <arg>-overwrite </arg>
<arg>${overwrite}</arg>
<arg>-ignoreErrors </arg>
<arg>${ignoreErrors}</arg>
<arg>-skipChecksum </arg>
<arg>${skipChecksum}</arg>
<arg>-removeDeletedFiles </arg>
<arg>${removeDeletedFiles}</arg>
<arg>-preserveBlockSize </arg>
<arg>${preserveBlockSize}</arg>
<arg>-preserveReplicationNumber </arg>
<arg>${preserveReplicationNumber}</arg>
<arg>-preservePermission </arg>
<arg>${preservePermission}</arg>
Pass below options in hdfs-replication.properties overwrite=false
ignoreErrors=false
skipChecksum=false
removeDeletedFiles=true
preserveBlockSize=true
preserveReplicationNumber=true
preservePermission=true
These will work OOTB as FeedReplicator already has support for this and hence no code change is required. Thanks!
... View more
