question Re: Falcon with HA Resource Manager in Archives of Support Questions (Read Only)

Falcon with HA Resource Manager

bleonhardi — Tue, 12 Jan 2016 01:29:40 GMT

I have seen the question for HA Namenodes however HA Resource Managers still confuse me. In Hue you are for example told to add a second resource manager entry with the same logical hue name. I.e. Hue supports adding two resource manager urls and he will manually try both.

How does that work in Falcon, how can I enter an HA Resource Manager entry into the interfaces of the cluster Entity document. For Namenode HA I would use the logical name and the program would then read the hdfs-site.xml

I have seen the other similar questions for oozie but I am not sure it was answered or I didn't really understand it.

https://community.hortonworks.com/questions/2740/what-value-should-i-use-for-jobtracker-for-resourc.html

so assuming my active resource manager is

mycluster1.com:8050

and standby is

mycluster2,com:8050

Re: Falcon with HA Resource Manager

aervits — Tue, 12 Jan 2016 01:31:44 GMT

@Benjamin Leonhard

RM works on the premise that even if you specify the standby RM URL, it will know to redirect to the active RM.

Re: Falcon with HA Resource Manager

bleonhardi — Tue, 12 Jan 2016 01:32:51 GMT

ah nice undercover magic. I will try and see what happens if I switch the active off.

Re: Falcon with HA Resource Manager

aervits — Tue, 12 Jan 2016 01:34:25 GMT

Client, ApplicationMaster and NodeManager on RM failover

When there are multiple RMs, the configuration (yarn-site.xml) used by clients and nodes is expected to list all the RMs. Clients, ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the RMs in a round-robin fashion until they hit the Active RM. If the Active goes down, they resume the round-robin polling until they hit the “new” Active. This default retry logic is implemented as org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider. You can override the logic by implementing org.apache.hadoop.yarn.client.RMFailoverProxyProvider and setting the value of yarn.client.failover-proxy-provider to the class name.

here's more info. @Benjamin Leonhardi

Re: Falcon with HA Resource Manager

aervits — Wed, 03 Feb 2016 05:17:40 GMT

@Benjamin Leonhardi are you still having problems with this? Can you provide your own solution or accept best answer?

Re: Falcon with HA Resource Manager

sunile_manjee — Mon, 11 Jul 2016 11:54:58 GMT

@Benjamin Leonhardi did this work as expected? if we simply select one of the RMs in falcon, will it fail over automatically to secondary? I am trying to understand the impact if Falcon pointing to RM and that RM goes down.

Re: Falcon with HA Resource Manager

bleonhardi — Mon, 11 Jul 2016 16:50:44 GMT

@Sunile Manjee yeah it works Alex tested it as well. Falcon does not point to one RM. It goes to the yarn-site.xml and finds the RM couple that has one of the ones you specified. Then it tries both.