Oozie job: What value should I use for jobTracker for Resource Manager HA?

[#1]

I was always wondering what value I should use for "jobtTacker" in my job.properites for Resource Manager HA.

Now a customer asked same question, so I thought this might be a good opportunity to find out.

Does anyone know which string we should use to utilize YARN Resource Manager HA?

According to Google, Cloudera uses "logicaljt" but I don't see this string in HDP code so far.

[#2 · Accepted solution]

@Hajime

Useful information. Link

When there are multiple RMs, the configuration (yarn-site.xml) used by clients and nodes is expected to list all the RMs. Clients, ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the RMs in a round-robin fashion until they hit the Active RM. If the Active goes down, they resume the round-robin polling until they hit the “new” Active. This default retry logic is implemented as org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider. You can override the logic by implementing org.apache.hadoop.yarn.client.RMFailoverProxyProvider and setting the value of yarn.client.failover-proxy-provider to the class name.

Editing based on the discussion.

Oozie and RM HA

[#3]

But how about "jobTracker" in job.properties? I need to type "jobTrackers=hostname:8032"

[#4]

@Hajime What version of HDP are you using?

[#5]

i'm using 2.3.2 (but customer is 2.2.4.2)

[#6]

@[email protected] This is something we need to look into.

[#7]

I believe you can specify one of the nodes, if it's not the active RM, API will detect the proper RM and redirect.

[#8]

@Hajime @[email protected]

Based on this discussion with Hajime "this is what we have in our repo. Credit to Hajime for finding that link"

[#9]

@Hajime usually you pick one of the nodes for RM as far as Oozie is concerned. If you issue to a standby node, it will round-robin to the active RM.

[#10]

This is the hortonworks docs that gives reference to it.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_hadoop-ha/content/deploy-hue-rm-ha-clust...