question Re: oozie shell action in Support Questions

oozie shell action

vtpcnk — Tue, 05 Mar 2019 22:54:34 GMT

On execution the yarn log shows as success.

But I can't see the /tmp/output file being created anywhere. I checked on the local linux host as well as hdfs.

One question I have is in job.properties -

oozie.coord.application.path=${nameNode}/user/root/apps/shell

Should the configuration files (job.properties, coordinator.xml etc) reside in the above directory? I have them there.

Not sure what is happening.

Appreciate the insights.

Re: oozie shell action

vtpcnk — Wed, 06 Mar 2019 02:18:23 GMT

In the yarn log for the job I can see :

=================================================================

>>> Invoking Shell command line now >>

Exit code of the Shell command 0
<<< Invocation of Shell command completed <<<

<<< Invocation of Main class completed <<<

Oozie Launcher, capturing output data:
=======================

Re: oozie shell action

KuldeepK — Thu, 07 Mar 2019 03:01:39 GMT

Hi @n c

Can you please check on the node manager where this launcher was run, ideally this file should be created locally on the node manager where your shell script was run.

Please do let me know if you need any further help.

Re: oozie shell action

vtpcnk — Thu, 07 Mar 2019 03:01:40 GMT

@Kuldeep Kulkarni , any insights pls? Appreciate the help.

Re: oozie shell action

vtpcnk — Thu, 07 Mar 2019 03:10:17 GMT

>this file should be created locally on the node manager where your shell script was run

I have the file locally on a node. It is on this node that I execute the oozie command.

Are you saying that the shell script should be in hdfs? And that yarn will execute this shell script on the node manager on which this job was run? Rather than on the host from which I run the oozie command?

Appreciate the feedback.

Re: oozie shell action

vtpcnk — Thu, 07 Mar 2019 03:21:37 GMT

yes, I checked the nodes and found the output in one of them. I reran to make sure.

So the script is not really needed on the Linux box? all of them - job.properties, the shellscript, workflow.xml, coordinator.xml? They need to be only in hdfs?

Also next, how to execute a python code from oozie? Also I want it to run daily.

Appreciate the insights.

Re: oozie shell action

vtpcnk — Fri, 08 Mar 2019 22:34:18 GMT

@Kuldeep Kulkarni, I created python action based on https://community.hortonworks.com/content/supportkb/151119/how-to-run-a-python-script-using-oozie-shell-actio.html

But how do I integrate coordinator.xml with that?

I tried creating the file but it is not executing as per that.

Is there somewhere in job.properties or workflow.xml that you mention coordinator.xml?

Appreciate the feedback.

Re: oozie shell action

KuldeepK — Sat, 09 Mar 2019 02:33:44 GMT

@n c

job.properties -> Needs to be on local filesystem on Oozie client from where you wish to submit the oozie job -run command. No need to have this file on HDFS.

workflow.xml -> This needs to be on HDFS.

script -> This needs to be on HDFS.

Regarding your question on python script - Please refer below article for how to create coordinator ( Please ignore input event part )

https://community.hortonworks.com/articles/27497/oozie-coordinator-and-based-on-input-data-events.html

in coordinator.xml you mention "<app-path>${workflowAppUri}</app-path>" that's what is location of workflow.xml

Please use Ambari workflow manager to design coordinators and workflows with easy WebUI.

Hope this helps.

Please let me know if you have any questions. Please accept my answer if it was helpful. 🙂

Re: oozie shell action

vtpcnk — Sat, 09 Mar 2019 02:58:16 GMT

Hi Kuldeep, thanks so much for the clarification.

I will try to do as per your instructions and let you know how it went.

Thanks again.

Re: oozie shell action

vtpcnk — Sat, 09 Mar 2019 02:59:19 GMT

BTW now I use cloudera. So I guess it has to be manual. Thanks.

Re: oozie shell action

vtpcnk — Sat, 09 Mar 2019 03:07:53 GMT

@Kuldeep Kulkarni , does your example : https://community.hortonworks.com/articles/27497/oozie-coordinator-and-based-on-input-data-events.html

set the job to run once a day?

If not, can you please let me know how to do that? I want to run a job once daily. Thanks.

Re: oozie shell action

KuldeepK — Sat, 09 Mar 2019 03:38:41 GMT

@n c
For some reason, I'm not able to add a comment in this post so replying as an answer:

For CDH, you can use Hue to design the submit the workflow.

For frequency - here is you can change the frequency of coordinator as per your requirement

 frequency="${coord:days(1)}"

Re: oozie shell action

vtpcnk — Mon, 11 Mar 2019 21:02:23 GMT

@Kuldeep Kulkarni, there are many lines with "input data" in the page you referred - not sure which ones to ignore.

Should I ignore the sections for datasets/input events/output events - that will leave only the workflow section.

Is that right?

Can't I use the coordinator from your shell action example?

But in that I don't see : "<app-path>${workflowAppUri}</app-path>"

Appreciate the clarification.