Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to run a oozie shell action in YARN

How to run a oozie shell action in YARN

Explorer

Hi,

 

How to run a oozie shell action in YARN (normally we provide JT and NN information via job.properties but what to provide in YARN).

 

Can someone please advice me, currently we are using YARN and we have enable mapreduce service only to run the oozie because I am not sure what information I should provide in job.properties if I am using YARN.

 

Currently all the job logs is re-directed to job-tracker:50030. But if we are running any mapreduce third party job then it's going to resource manager port:19888/jobhistory.

 

Hence if I am running any job via oozie it is running in MVr1 whereas other third party MR jobs are running in YARN (I can see in 19888/jobhistory).

 

Please advice how I can set-up the oozie to run in YARN.

 
5 REPLIES 5

Re: How to run a oozie shell action in YARN

Master Guru
You don't need to do much other than specify the RM host and port (or
HA URI if you use RM HA) in place of the JobTracker variable in the
submission properties.

However, to make Oozie first switch to YARN you need to follow these
steps documented at
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_mc_mr_and_yarn.html?...

"""
To update a service to use YARN:

Go to the service (such as Oozie).
Click the Configuration tab.
Select Service-Wide.
Click the MapReduce Service property and select the YARN or MapReduce service.
Click Save Changes to commit the changes.
Select Actions > Restart.
"""

After this, also perform an reinstallation of Oozie Sharelib from CM
-> Oozie -> Actions -> Install ShareLib… button (you may need to stop
Oozie again before this is activated, depending on your CM version).

Does this help?

Re: How to run a oozie shell action in YARN

Explorer
Hello Harsh,

Thanks for your solution, if you could provide me the sample job.properties
and a xml for YARN then it would be a great help.
Little new to oozie.

Re: How to run a oozie shell action in YARN

Master Guru
There's no XML changes required. In your job.properties, just change
the JT URL to the RM one:

For ex., if you have:

jobTracker=localhost:8021

Just make it:

jobTracker=localhost:8032

If you instead have YARN HA enabled, then it should just be the system
ID, for example:

jobTracker=yarnRM

(Assuming localhost is the hostname where both JT and RM run, so
change as per your environment…)

Re: How to run a oozie shell action in YARN

Explorer

It's thowing me a error.

AUTHENTICATION : Could not authenticate, Authentication failed, status: 404, message: Not Found

I have done the first step what you have suggested. And then I have tried to install oozie share lid but it failed. again started the oozie , in job.properties file made localhost:8032

Highlighted

Re: How to run a oozie shell action in YARN

Explorer

Can someone help me here, I am not able to install oozie shared lib. The below logs are there for your referece.

 

Failed to upload Oozie ShareLib.

 

AuthenticationToken ignored: AuthenticationToken expired
Error: Mkdirs failed to create /tmp/oozie6599766817659088664.dir

Stack trace for the error was (for debug purposes):
--------------------------------------
java.io.IOException: Mkdirs failed to create /tmp/oozie6599766817659088664.dir
	at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:638)
	at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:130)
	at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:56)