question Re: How to run Oozie Job with Python Script in Sandbox? in Archives of Support Questions (Read Only)

How to run Oozie Job with Python Script in Sandbox?

axel_robinson — Wed, 04 Jan 2017 01:56:45 GMT

workflow.xml Attached is my current xml file in the sandbox. I am new to Hadoop/Oozie so any detail explanation/instruction would be very helpful

Re: How to run Oozie Job with Python Script in Sandbox?

aervits — Wed, 04 Jan 2017 08:55:54 GMT

I am mobile and can't comment on your workflows right now but I have example of python2 and python3 WF in my repo https://github.com/dbist/oozie

Browse to oozie/apps/ and you will see their respective directories. Use as you wish.

Re: How to run Oozie Job with Python Script in Sandbox?

axel_robinson — Thu, 05 Jan 2017 00:39:12 GMT

Thank you for your prompt reply. But I have a very basic question. Python should be install in the Sandbox? apps-directory.png Currently my python scripts are placed in /root/examples/apps/map-reduce/ But I am guessing there should be a python folder in root/examples/apps/Python? which contains the job.properties and workflow.xml files in addition to the lib folder would be great if you could guide in which directory should I place the python script files

Re: How to run Oozie Job with Python Script in Sandbox?

aervits — Thu, 05 Jan 2017 04:07:48 GMT

@justlearning same version of Python needs to be installed on every node that will run oozie containers (nodemanager). Same goes for any Python libraries you're importing into your script. I usually create the following tree

admin@u1201:~/oozie/apps/python$ tree
.
|-- job.properties
|-- scripts
|   `-- script.py
`-- workflow.xml


1 directory, 3 files

so what you want is a workflow directory on hdfs with at least workflow.xml and optionally another directory within it with a Python script. job.properties file needs to be on your local filesystem. Then you would execute the oozie wf the following way:

oozie job -oozie http://u1203.ambari.apache.org:11000/oozie -config oozie/apps/python/job.properties -run

Re: How to run Oozie Job with Python Script in Sandbox?

axel_robinson — Fri, 06 Jan 2017 02:53:03 GMT

@Artem Ervits ohh thank you so much .. How do you determine where the output of the job should be store/ how can you see the output to be sure it was what you're looking for

Re: How to run Oozie Job with Python Script in Sandbox?

aervits — Fri, 06 Jan 2017 21:05:56 GMT

You can learn that by following this guide https://community.hortonworks.com/content/kbentry/9148/troubleshooting-an-oozie-flow.html

Re: How to run Oozie Job with Python Script in Sandbox?

aervits — Sat, 07 Jan 2017 06:37:26 GMT

It's a good practice to accept answer if it satisfies your needs.