question Oozie job: What value should I use for jobTracker for Resource Manager HA? in Support Questions

Oozie job: What value should I use for jobTracker for Resource Manager HA?

hosako — Wed, 04 Nov 2015 08:53:29 GMT

I was always wondering what value I should use for "jobtTacker" in my job.properites for Resource Manager HA.

Now a customer asked same question, so I thought this might be a good opportunity to find out.

Does anyone know which string we should use to utilize YARN Resource Manager HA?

According to Google, Cloudera uses "logicaljt" but I don't see this string in HDP code so far.

Re: Oozie job: What value should I use for jobTracker for Resource Manager HA?

nsabharwal — Wed, 04 Nov 2015 08:58:23 GMT

@Hajime

Useful information. Link

When there are multiple RMs, the configuration (yarn-site.xml) used by clients and nodes is expected to list all the RMs. Clients, ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the RMs in a round-robin fashion until they hit the Active RM. If the Active goes down, they resume the round-robin polling until they hit the “new” Active. This default retry logic is implemented as org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider. You can override the logic by implementing org.apache.hadoop.yarn.client.RMFailoverProxyProvider and setting the value of yarn.client.failover-proxy-provider to the class name.

Editing based on the discussion.

Oozie and RM HA

Re: Oozie job: What value should I use for jobTracker for Resource Manager HA?

hosako — Wed, 04 Nov 2015 09:00:38 GMT

But how about "jobTracker" in job.properties? I need to type "jobTrackers=hostname:8032"

Re: Oozie job: What value should I use for jobTracker for Resource Manager HA?

nsabharwal — Wed, 04 Nov 2015 09:03:00 GMT

@Hajime What version of HDP are you using?

Re: Oozie job: What value should I use for jobTracker for Resource Manager HA?

hosako — Wed, 04 Nov 2015 09:06:11 GMT

i'm using 2.3.2 (but customer is 2.2.4.2)

Re: Oozie job: What value should I use for jobTracker for Resource Manager HA?

nsabharwal — Wed, 04 Nov 2015 09:39:49 GMT

@jeff@hortonworks.com This is something we need to look into.

Re: Oozie job: What value should I use for jobTracker for Resource Manager HA?

nsabharwal — Wed, 04 Nov 2015 21:32:16 GMT

@Hajime @jeff@hortonworks.com

Based on this discussion with Hajime "this is what we have in our repo. Credit to Hajime for finding that link"

Re: Oozie job: What value should I use for jobTracker for Resource Manager HA?

aervits — Thu, 05 Nov 2015 00:39:23 GMT

I believe you can specify one of the nodes, if it's not the active RM, API will detect the proper RM and redirect.

Re: Oozie job: What value should I use for jobTracker for Resource Manager HA?

aervits — Wed, 03 Feb 2016 09:57:41 GMT

@Hajime usually you pick one of the nodes for RM as far as Oozie is concerned. If you issue to a standby node, it will round-robin to the active RM.

Re: Oozie job: What value should I use for jobTracker for Resource Manager HA?

sdutta — Wed, 04 May 2016 08:17:39 GMT

This is the hortonworks docs that gives reference to it.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_hadoop-ha/content/deploy-hue-rm-ha-cluster.html

Re: Oozie job: What value should I use for jobTracker for Resource Manager HA?

phil1 — Wed, 20 Jul 2016 15:33:26 GMT

In Oozie, redirection may work at the job level but I have another issue, namely that the Oozie WebUI shows nothing for action info when say, "other" RM is active.

For some reason, either redirection is not occuring there or there is a configuration to set in Oozie WebUI that makes that redirection happen.

HDP 2.3 here.

Clues?