Before creating the cluster entities, we need to create the directories on HDFS representing the cluster that we are going to define, namely primaryCluster in your case. su - falcon hadoop fs -mkdir /apps/falcon/primaryCluster Further create directories called staging and working hadoop fs -mkdir /apps/falcon/primaryCluster/staging hadoop fs -mkdir /apps/falcon/primaryCluster/working Finally you need to set the proper permissions on the staging/working directories: hadoop fs -chmod 777 /apps/falcon/primaryCluster/staging hadoop fs -chmod 755 /apps/falcon/primaryCluster/working hadoop fs -chown -R falcon /apps/falcon/* You can refer to http://hortonworks.com/hadoop-tutorial/processing-data-pipeline-with-apache-falcon/ for more details. ... View more

Falcon can be installed on source or destination cluster. By design it can also run as a stand alone server [not part of any cluster]. There is no requirement that it has to be installed on both source and target cluster. In Falcon today, replication can be achieved using 1> Feed replication : By default its a pull mechanism., replication job runs on target cluster. Falcon takes jobTracker and nameNode properties from target cluster entity execute and write end points resp. and submits the replication job to Oozie using the workflow endpoint specified in target cluster entity - this ensures the replication job runs on target cluster. 2> Mirroring using recipes: For recipes the user can configure the job to run on source or target cluster. If you look at the Mirroring falcon UI there is a option button "Run job here" for source and target. Default setting is ""Run job here" for target - so by default for HDFS mirroring or Hive mirroring, replication job runs on target cluster. Design decision to use pull mechanism i.e. to run replication job on target cluster was to avoid class path issues if clusters use different Hadoop versions and other reasons. So running job on source cluster may not work always but user has the flexibility to do so. For cases where one of the cluster involved in replication is a non Hadoop cluster say S3/ Azure, then replication job always run on Hadoop cluster. Regarding Falcon prism/distributed mode Apache Falcon has this feature and InMobi uses it in their prod environment. This feature is not available in HDP. ... View more

If you are asking about using recipes for mirroring, if you look at hdfs-replication-template.xml or hive-replication-template.xml has below properties set to ensure only one instance runs. <parallel>1</parallel> <!-- Dir replication needs to run only once to catch up --> <order>LAST_ONLY</order> -------------------------------------------------------------------------------------------------------------- In general, in Falcon this can be controlled using <parallel> and <order> properties in the entity xml. Parallel decides the concurrent replication instances that can run at any given time and order decides the execution order for replication instances like FIFO, LIFO and LAST_ONLY. For replication job needs to run only once to catch up. Setting below configs in entity xml will ensure this: <parallel>1</parallel> <order>LAST_ONLY</order> Parallel maps to concurrency and order maps to execution in Oozie. Please refer Oozie doc for more details. ... View more

Today replication in Falcon can be achieved using two ways: 1> Feed based Replication: Falcon uses pull based replication mechanism, meaning in every target cluster, for a given source cluster, a coordinator is scheduled which pulls the data using DistCp from source cluster. This requires data locations to be replicated to have dated partitions. 2> Using concept of Recipes: HDFS Directory Replication Recipe Overview This recipe implements replicating arbitrary directories on HDFS from one Hadoop cluster to another Hadoop cluster. This piggy backs on replication solution in Falcon which uses the DistCp tool. Use Case * Copy directories between HDFS clusters with out dated partitions * Archive directories from HDFS to Cloud. Ex: S3, Azure WASB Limitations As the data volume and number of files grow, this can get inefficient. User should make sure data already replicated is evicted else it will have performance issues. For both of the above mechanisms, DistCp options can be passed as custom properties, which will be propagated to the DistCp tool. maxMaps represents the maximum number of maps used during replication mapBandwidth represents the bandwidth in MB/s used by each mapper during replication overwrite represents overwrite destination during replication ignoreErrors represents ignore failures not causing the job to fail during replication skipChecksum represents bypassing checksum verification during replication removeDeletedFiles represents deleting the files existing in the destination but not in source during replication preserveBlockSize represents preserving block size during replication preserveReplicationNumber represents preserving replication number during replication preservePermission represents preserving permission during replication ... View more

Yes, webHDFS is supported by falcon. ... View more