Hello, I'm working with Falcon using the built-in HDFS mirroring capabilities and would like to enable two distcp options in the workflow XML: the -atomic flag and -strategy flags. Below is my Oozie workflow with these two options commented out, as this approach was unsuccessful. Is there a way to pass these in using a -D option or would the FeedReplicator class need to be modified for this functionality? <workflow-app xmlns='uri:oozie:workflow:0.3' name='falcon-dr-fs-workflow'> <start to='dr-replication'/> <!-- Replication action --> <action name="dr-replication"> <java> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <!-- hadoop 2 parameter --> <name>oozie.launcher.mapreduce.job.user.classpath.first</name> <value>true</value> </property> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> <property> <name>oozie.launcher.mapred.job.priority</name> <value>${jobPriority}</value> </property> <property> <name>oozie.use.system.libpath</name> <value>true</value> </property> <property> <name>oozie.action.sharelib.for.java</name> <value>distcp</value> </property> <property> <name>oozie.launcher.oozie.libpath</name> <value>${wf:conf("falcon.libpath")}</value> </property> <property> <name>oozie.launcher.mapreduce.job.hdfs-servers</name> <value>${drSourceClusterFS},${drTargetClusterFS}</value> </property> </configuration> <main-class>org.apache.falcon.replication.FeedReplicator</main-class> <arg>-Dmapred.job.queue.name=${queueName}</arg> <arg>-Dmapred.job.priority=${jobPriority}</arg> <!--arg>-atomic</arg> <arg>-strategy</arg> <arg>dynamic</arg--> <arg>-maxMaps</arg> <arg>${distcpMaxMaps}</arg> <arg>-mapBandwidth</arg> <arg>${distcpMapBandwidth}</arg> <arg>-sourcePaths</arg> <arg>${drSourceDir}</arg> <arg>-targetPath</arg> <arg>${drTargetClusterFS}${drTargetDir}</arg> <arg>-falconFeedStorageType</arg> <arg>FILESYSTEM</arg> <arg>-availabilityFlag</arg> <arg>${availabilityFlag == 'NA' ? "NA" : availabilityFlag}</arg> <arg>-counterLogDir</arg> <arg>${logDir}/job-${nominalTime}/${srcClusterName == 'NA' ? '' : srcClusterName}</arg> </java> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message> Workflow action failed, error message[${wf:errorMessage(wf:lastErrorNode())}] </message> </kill> <end name="end"/> </workflow-app> ... View more

Falcon is going to drive me to an early grave. I've been able to create the Falcon replication job (hdfs-replication-workflow), define both the source and target clusters it uses. The process seems to start up in Falcon, two Oozie jobs also startup (see below), and some YARN jobs (see below) but can't seem to get the distcp portion to actually move data. I'm not sure if this is related to being in (two) sandbox environments or not but there isn't an obvious way to debug whats happening... most things either say "COMPLETED" or stay "RUNNING" forever - nothing noticeable is failing or creating helpful logs. Not sure if that is specific enough but Falcon itself is a can of worms that I don't totally understand yet... Oozie and YARN and dispersed logs... sorry, I'll stop complaining. Job ID App Name Status User Group Started Ended ------------------------------------------------------------------------------------------------------------------------------------ 0000067-160817182129092-oozie-oozi-W falcon-dr-fs-workflowRUNNING gpadmin - 2016-08-18 21:55 GMT - ------------------------------------------------------------------------------------------------------------------------------------ 0000066-160817182129092-oozie-oozi-W FALCON_PROCESS_DEFAULT_drSyncTestRUNNING gpadmin - 2016-08-18 21:55 GMT - ------------------------------------------------------------------------------------------------------------------------------------ Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):2 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1471553333960_0013 oozie:launcher:T=java:W=falcon-dr-fs-workflow:A=dr-replication:ID=0000071-160817182129092-oozie-ooz i-W MAPREDUCE gpadmin default RUNNING UNDEFINED 5% htt p://sandbox2.hortonworks.com:18800 application_1471553333960_0014 distcp: oozie:action:T=java:W=falcon-dr-fs-workflow:A=dr-replication:ID=0000071-160817182129092-ooz ie-oozi-W MAPREDUCE gpadmin default ACCEPTED UNDEFINED 0% N/A ... View more

I'm trying to migrate from Pivotal Hadoop to HDP without re-deploying the cluster. I've managed to get Ambari 2.x to migrate the Postgres database schema and assume control of the cluster services but when I navigate to the Ambari Stack/Version page (main/admin/stack/versions) the expected "version tiles" aren't listed. I've tried to reference a working HDP 2.4 cluster to see if there are database entries missing that are used to popular these fields but haven't been able to track them down. In the manage versions page (views/ADMIN_VIEW/2.2.2.0/INSTANCE/#/stackVersions) I do see both 2.2.9 and 2.4.2 repositories registered correctly. Any ideas? Thanks. ... View more