Falcon support on Hadoop HA services

[#1]

Where could I find a complete guide on Falcon configuration for HA services on both source and target clusters?

So for NN HA, the solution is clear - define both clusters in hdfs-site.xml in source and target clusters.

How about RM HA? With both RM1 and RM2 as active and passive, should I just pick one for source cluster such as in the following:

https://community.hortonworks.com/questions/9416/falcon-with-ha-resource-manager.html

<interface type="execute" endpoint="RM1:8050" version="2.2.0" />

What should I do for remote cluster RM? Just pick one?

How do we define "hive.metastore.kerberos.principal" and "hive.metastore.uris" for multiple Hive metastore instances?

How do we define for multiple oozie instances?

[#2 · Accepted solution]

@ScipioTheYounger

should I just pick one for source cluster such as in the following:

https://community.hortonworks.com/questions/9416/falcon-with-ha-resource-manager.html

<interface type="execute" endpoint="RM1:8050" version="2.2.0" />

Yes that is correct.

How do we define "hive.metastore.kerberos.principal"

Refer https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/configu...

[#3]

For replication across the clusters, Falcon uses pull mode which Ooize/RM on target cluster should be used. If the RM specified is DOWN, not standby, do we have any problem for Falcon?

[#4]

Client, ApplicationMaster and NodeManager on RM failover

When there are multiple RMs, the configuration (yarn-site.xml) used by clients and nodes is expected to list all the RMs. Clients, ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the RMs in a round-robin fashion until they hit the Active RM. If the Active goes down, they resume the round-robin polling until they hit the “new” Active. This default retry logic is implemented as org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider. You can override the logic by implementing org.apache.hadoop.yarn.client.RMFailoverProxyProvider and setting the value of yarn.client.failover-proxy-provider to the class name.

[#5]

Thanks Sunile. The replication Oozie WF is defined in target cluster, and WF will be run by RMs in target cluster, so there should be no problem then.

[#6]

What should I do for remote cluster RM? Just pick one?

Yes. You can use one of the RM endpoints

How do we define "hive.metastore.kerberos.principal" and "hive.metastore.uris" for multiple Hive metastore instances?

This is currently fixed in 2.5 and we can provide thrift://host1:port,host2:port

How do we define for multiple oozie instances?

You can use the LB URL