question Re: Falcon support on Hadoop HA services in Archives of Support Questions (Read Only)

Falcon support on Hadoop HA services

ScipioTheElder — Thu, 07 Jul 2016 02:45:00 GMT

Where could I find a complete guide on Falcon configuration for HA services on both source and target clusters?

So for NN HA, the solution is clear - define both clusters in hdfs-site.xml in source and target clusters.

How about RM HA? With both RM1 and RM2 as active and passive, should I just pick one for source cluster such as in the following:

https://community.hortonworks.com/questions/9416/falcon-with-ha-resource-manager.html

What should I do for remote cluster RM? Just pick one?

How do we define "hive.metastore.kerberos.principal" and "hive.metastore.uris" for multiple Hive metastore instances?

How do we define for multiple oozie instances?

Re: Falcon support on Hadoop HA services

rpathak — Thu, 07 Jul 2016 02:52:23 GMT

@ScipioTheYounger

should I just pick one for source cluster such as in the following:

https://community.hortonworks.com/questions/9416/falcon-with-ha-resource-manager.html

Yes that is correct.

How do we define "hive.metastore.kerberos.principal"

Refer https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/configuring_for_secure_clusters_falcon.html

Re: Falcon support on Hadoop HA services

vranganathan — Thu, 07 Jul 2016 04:26:00 GMT

What should I do for remote cluster RM? Just pick one?

Yes. You can use one of the RM endpoints

How do we define "hive.metastore.kerberos.principal" and "hive.metastore.uris" for multiple Hive metastore instances?

This is currently fixed in 2.5 and we can provide thrift://host1:port,host2:port

How do we define for multiple oozie instances?

You can use the LB URL

Re: Falcon support on Hadoop HA services

ScipioTheElder — Thu, 07 Jul 2016 21:57:43 GMT

For replication across the clusters, Falcon uses pull mode which Ooize/RM on target cluster should be used. If the RM specified is DOWN, not standby, do we have any problem for Falcon?

Re: Falcon support on Hadoop HA services

sunile_manjee — Mon, 11 Jul 2016 11:44:53 GMT

Client, ApplicationMaster and NodeManager on RM failover

When there are multiple RMs, the configuration (yarn-site.xml) used by clients and nodes is expected to list all the RMs. Clients, ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the RMs in a round-robin fashion until they hit the Active RM. If the Active goes down, they resume the round-robin polling until they hit the “new” Active. This default retry logic is implemented as org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider. You can override the logic by implementing org.apache.hadoop.yarn.client.RMFailoverProxyProvider and setting the value of yarn.client.failover-proxy-provider to the class name.

Re: Falcon support on Hadoop HA services

ScipioTheElder — Mon, 11 Jul 2016 22:28:11 GMT

Thanks Sunile. The replication Oozie WF is defined in target cluster, and WF will be run by RMs in target cluster, so there should be no problem then.