Support Questions
Find answers, ask questions, and share your expertise

Falcon support on Hadoop HA services

Solved Go to solution
Highlighted

Falcon support on Hadoop HA services

Contributor

Where could I find a complete guide on Falcon configuration for HA services on both source and target clusters?

So for NN HA, the solution is clear - define both clusters in hdfs-site.xml in source and target clusters.

How about RM HA? With both RM1 and RM2 as active and passive, should I just pick one for source cluster such as in the following:

https://community.hortonworks.com/questions/9416/falcon-with-ha-resource-manager.html

<interface type="execute" endpoint="RM1:8050" version="2.2.0" />

What should I do for remote cluster RM? Just pick one?

How do we define "hive.metastore.kerberos.principal" and "hive.metastore.uris" for multiple Hive metastore instances?

How do we define for multiple oozie instances?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Falcon support on Hadoop HA services

@ScipioTheYounger

should I just pick one for source cluster such as in the following:

https://community.hortonworks.com/questions/9416/falcon-with-ha-resource-manager.html

<interface type="execute" endpoint="RM1:8050" version="2.2.0" />

Yes that is correct.

How do we define "hive.metastore.kerberos.principal"

Refer https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/configu...

View solution in original post

5 REPLIES 5
Highlighted

Re: Falcon support on Hadoop HA services

@ScipioTheYounger

should I just pick one for source cluster such as in the following:

https://community.hortonworks.com/questions/9416/falcon-with-ha-resource-manager.html

<interface type="execute" endpoint="RM1:8050" version="2.2.0" />

Yes that is correct.

How do we define "hive.metastore.kerberos.principal"

Refer https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/configu...

View solution in original post

Highlighted

Re: Falcon support on Hadoop HA services

Contributor

For replication across the clusters, Falcon uses pull mode which Ooize/RM on target cluster should be used. If the RM specified is DOWN, not standby, do we have any problem for Falcon?

Highlighted

Re: Falcon support on Hadoop HA services

Super Guru

Client, ApplicationMaster and NodeManager on RM failover

When there are multiple RMs, the configuration (yarn-site.xml) used by clients and nodes is expected to list all the RMs. Clients, ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the RMs in a round-robin fashion until they hit the Active RM. If the Active goes down, they resume the round-robin polling until they hit the “new” Active. This default retry logic is implemented as org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider. You can override the logic by implementing org.apache.hadoop.yarn.client.RMFailoverProxyProvider and setting the value of yarn.client.failover-proxy-provider to the class name.

Highlighted

Re: Falcon support on Hadoop HA services

Contributor

Thanks Sunile. The replication Oozie WF is defined in target cluster, and WF will be run by RMs in target cluster, so there should be no problem then.

Re: Falcon support on Hadoop HA services

Rising Star

What should I do for remote cluster RM? Just pick one?

Yes. You can use one of the RM endpoints

How do we define "hive.metastore.kerberos.principal" and "hive.metastore.uris" for multiple Hive metastore instances?

This is currently fixed in 2.5 and we can provide thrift://host1:port,host2:port

How do we define for multiple oozie instances?

You can use the LB URL