Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to point a WorkFlow manager job to a certain python envirnment ?

Highlighted

How to point a WorkFlow manager job to a certain python envirnment ?

Explorer

How to point a job in workflow manager to a specific python envirnment?

5 REPLIES 5
Highlighted

Re: How to point a WorkFlow manager job to a certain python envirnment ?

@Victor Try to execute python binary directly with full path (for example /opt/anaconda3/bin/python3) and pass the script .py file as argument. Make sure anaconda3 is installed on all the cluster nodes.

HTH

Highlighted

Re: How to point a WorkFlow manager job to a certain python envirnment ?

Explorer

@Felix Albani

 Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

This is the head of the python file

 #!/opt/anaconda3/envs/pythonoozie/bin python3.6

And that is because it could not import pandas.
I can see that althought I use the /opt/anaconda3/bin/python which through the shell has pandas and is python3, oozie somehow manages to do the python2.6 which does not have pandas.

77631-z1.png

77632-z2.png

77633-z3.png

77634-xml.png

Highlighted

Re: How to point a WorkFlow manager job to a certain python envirnment ?

Explorer

@Felix Albani Any updates on this issue :) Thank you

Highlighted

Re: How to point a WorkFlow manager job to a certain python envirnment ?

@Victor Perhaps you can use a shell script instead of calling the python directly. There you can set all the necessary environment variables and install the necessary library dependencies if not installed yet.

HTH

Re: How to point a WorkFlow manager job to a certain python envirnment ?

Explorer
@Felix Albani

It seems like Oozie has all the variables to the python set by default to Python 2.6.6 in the usr/ directory. The best way to approach this is to find a way to change the default folder of where oozie points to python. How can I find the configuration in Oozie that points to the Python directory? and then override it with Anaconda virtual environment ?

Don't have an account?
Coming from Hortonworks? Activate your account here