question Re: How do you specify a highly-available HDFS namespace in an Apache Falcon cluster definition in Support Questions

How do you specify a highly-available HDFS namespace in an Apache Falcon cluster definition

dkjerrumgaard1 — Tue, 08 Dec 2015 07:04:34 GMT

for the following interfaces:

If we are pointing to a cluster with HDFS H/A enabled?

Re: How do you specify a highly-available HDFS namespace in an Apache Falcon cluster definition

bvellanki — Tue, 08 Dec 2015 07:19:48 GMT

If the nameservice is "myHA", the interfaces should be "hdfs://myHA".

Re: How do you specify a highly-available HDFS namespace in an Apache Falcon cluster definition

dkjerrumgaard1 — Wed, 09 Dec 2015 00:30:17 GMT

Just to clarify, the cluster in question is different from the one where Falcon is running, i.e. it is a D/R cluster we want to copy data to..

Re: How do you specify a highly-available HDFS namespace in an Apache Falcon cluster definition

landon_t_robins — Wed, 09 Dec 2015 23:50:11 GMT

You can point to it directly via its address, or you can do as @bvellanki (balu) mentioned, and list its HA. For example, if your HA for your backup cluster is called DRHA, your address would be hdfs://DRHA:8020. See below:

<interface type="readonly" endpoint="hftp://DRHA.company.com:50070" version="2.2.0"/>         
<interface type="write" endpoint="hdfs://DRHA.company.com:8020" version="2.2.0"/> 

#You can also do this, depending on preference

<interface type="readonly" endpoint="hftp://DRHA:50070" version="2.2.0"/>         
<interface type="write" endpoint="hdfs://DRHA:8020" version="2.2.0"/>

Re: How do you specify a highly-available HDFS namespace in an Apache Falcon cluster definition

mmiklavcic — Thu, 10 Dec 2015 02:54:33 GMT

There are a couple considerations that need to be taken into account when using NN HA with Falcon and Oozie. In all cases, you need to use the Namenode service ID when referring to the Namenode in the cluster xml. This value can be found in hdfs-site.xml in the property dfs.ha.namenodes.[nameservice ID]. For multi-cluster installs, you need to setup all cluster Namenode HA nameservice ID details in all clusters. For example, if you have two clusters, hdfs-site.xml for both cluster one and cluster two will have 2 nameservice IDs. Likewise, for three clusters, all three clusters would have three nameservice IDs. A two-cluster implementation would look similar to the following:

<property>
  <name>dfs.ha.namenodes.hacluster1</name>
  <value>c1nn1,c1nn2</value>
</property>
<property>
  <name>dfs.ha.namenodes.hacluster2</name>
  <value>c2nn1,c2nn2</value>
</property>

Now, when you setup Falcon, provide both cluster definitions on both clusters.

Re: How do you specify a highly-available HDFS namespace in an Apache Falcon cluster definition

aervits — Tue, 02 Feb 2016 22:35:16 GMT

@dkjerrumgaard has this been resolved? Can you post your solution or accept best answer?