question Re: Falcon HDFS mirror distcp options in Support Questions

Falcon HDFS mirror distcp options

kdunn926 — Fri, 02 Sep 2016 01:19:05 GMT

Hello,

I'm working with Falcon using the built-in HDFS mirroring capabilities and would like to enable two distcp options in the workflow XML: the -atomic flag and -strategy flags. Below is my Oozie workflow with these two options commented out, as this approach was unsuccessful. Is there a way to pass these in using a -D option or would the FeedReplicator class need to be modified for this functionality?

<workflow-app xmlns='uri:oozie:workflow:0.3' name='falcon-dr-fs-workflow'>
    <start to='dr-replication'/>
    <!-- Replication action -->
    <action name="dr-replication">
        <java>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property> <!-- hadoop 2 parameter -->
                    <name>oozie.launcher.mapreduce.job.user.classpath.first</name>
                    <value>true</value>
                </property>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>oozie.launcher.mapred.job.priority</name>
                    <value>${jobPriority}</value>
                </property>
                <property>
                    <name>oozie.use.system.libpath</name>
                    <value>true</value>
                </property>
                <property>
                    <name>oozie.action.sharelib.for.java</name>
                    <value>distcp</value>
                </property>
                <property>
                    <name>oozie.launcher.oozie.libpath</name>
                    <value>${wf:conf("falcon.libpath")}</value>
                </property>
                <property>
                    <name>oozie.launcher.mapreduce.job.hdfs-servers</name>
                    <value>${drSourceClusterFS},${drTargetClusterFS}</value>
                </property>
            </configuration>
            <main-class>org.apache.falcon.replication.FeedReplicator</main-class>
            <arg>-Dmapred.job.queue.name=${queueName}</arg>
            <arg>-Dmapred.job.priority=${jobPriority}</arg>

            <!--arg>-atomic</arg>
            <arg>-strategy</arg>
            <arg>dynamic</arg-->

            <arg>-maxMaps</arg>
            <arg>${distcpMaxMaps}</arg>
            <arg>-mapBandwidth</arg>
            <arg>${distcpMapBandwidth}</arg>
            <arg>-sourcePaths</arg>
            <arg>${drSourceDir}</arg>
            <arg>-targetPath</arg>
            <arg>${drTargetClusterFS}${drTargetDir}</arg>
            <arg>-falconFeedStorageType</arg>
            <arg>FILESYSTEM</arg>
            <arg>-availabilityFlag</arg>
            <arg>${availabilityFlag == 'NA' ? "NA" : availabilityFlag}</arg>
            <arg>-counterLogDir</arg>
            <arg>${logDir}/job-${nominalTime}/${srcClusterName == 'NA' ? '' : srcClusterName}</arg>
        </java>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>
            Workflow action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
        </message>
    </kill>
    <end name="end"/>
</workflow-app>

Re: Falcon HDFS mirror distcp options

sramesh — Fri, 02 Sep 2016 01:55:50 GMT

@Kyle Dunn: Falcon doesn't support those DistCP options and yes that would require a code change.

Re: Falcon HDFS mirror distcp options

kdunn926 — Fri, 02 Sep 2016 01:55:51 GMT

Would you be able to provide an example of what this code change might be similar to in the existing FeedReplicator code?

Re: Falcon HDFS mirror distcp options

kdunn926 — Fri, 02 Sep 2016 06:16:45 GMT

Alternatively, what are the limitations of out-of-stack version support for Falcon? The snapshot-based replication in Falcon 0.10 provides the ultimate functionality I'm looking for, but am currently running on HDP 2.3 / 2.4.

Re: Falcon HDFS mirror distcp options

sramesh — Fri, 02 Sep 2016 12:17:33 GMT

What limitations are we talking about here? Sorry, I don't understand your question.

If you are asking about DIstCP options supported in HDFS Mirroirng, currently below options are supported

maxMaps
mapBandwidth

Below additional options can be supported by using workaround given below:

overwrite
ignoreErrors
skipChecksum
removeDeletedFiles
preserveBlockSize
preserveReplicationNumber
preservePermission

Please modify the WF hdfs-replication-workflow.xml as below. After distcpMapBandwidth add below content

<arg>-overwrite </arg>
<arg>${overwrite}</arg>
<arg>-ignoreErrors </arg>
<arg>${ignoreErrors}</arg>
<arg>-skipChecksum </arg>
<arg>${skipChecksum}</arg>
<arg>-removeDeletedFiles </arg>
<arg>${removeDeletedFiles}</arg>
<arg>-preserveBlockSize </arg>
<arg>${preserveBlockSize}</arg>
<arg>-preserveReplicationNumber </arg>
<arg>${preserveReplicationNumber}</arg>
<arg>-preservePermission </arg>
<arg>${preservePermission}</arg>

Pass below options in hdfs-replication.properties

overwrite=false
ignoreErrors=false
skipChecksum=false
removeDeletedFiles=true
preserveBlockSize=true
preserveReplicationNumber=true
preservePermission=true

These will work OOTB as FeedReplicator already has support for this and hence no code change is required. Thanks!

Re: Falcon HDFS mirror distcp options

Kyivstar — Tue, 14 Nov 2017 15:57:42 GMT

Hi!

After change parameter preserveBlockSize & skipChecksum on target site,  do not see any change in xml file for  task (after recreate task) :

[hdfs@target ~]$ hdfs dfs -ls /apps/falcon/extensions/hdfs-mirroring/retargets/runtime/hdfs-mirroring-workflow.xml
-rwxr-xr-x   2 hdfs users       4943 2017-11-13 22:39 /apps/falcon/extensions/hdfs-mirroring/retargets/runtime/hdfs-mirroring-workflow.xml        <<<  change  this file ( on target size)
[hdfs@target ~]$
t1.xml[hdfs@target ~]$ grep -i preserveBlockSize hdfs-mirroring-workflow.xml
            <arg>-preserveBlockSize</arg>
            <arg>${preserveBlockSize}</arg>
            <arg>-preserveBlockSize</arg> <arg>true</arg>
[hdfs@target ~]$
[hdfs@target ~]$
[hdfs@target ~]$ grep -i skipChecksum hdfs-mirroring-workflow.xml
            <arg>-skipChecksum</arg>
            <arg>${skipChecksum}</arg>
            <arg>-skipChecksum</arg> <arg>true</arg>
[hdfs@target ~]$

Please help me.

Where I can find file hdfs-replication.properties ?