Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Conda Environment in Nifi

Solved Go to solution
Highlighted

Conda Environment in Nifi

I have python project(have multiple script files and one main script file which triggers other script files). I can able to execute the project in my local machine. I activated the conda environment in my local machine from there the project is picking all the required dependencies(there are lot of dependencies both native and non-native) and executing the project. Now I want execute the same project in Nifi. As the project have the dependency modules not sure how to pick those modules as part of the Nifi flow(With Conda env or as part of path).

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Conda Environment in Nifi

The short answer is that you cannot embed such an environment to work within NiFi, but you can use NiFi to make calls against it.
NiFi embeds Jython, not full Python, so you cannot realistically get it to embed a full Anaconda environment. Typically what I do is use the ExecuteStreamCommand to call a small bash script which activates the python virtualenv and runs the script against the data, then returns to NiFi to carry on the flow.
Give this a try and let me know how you go.

View solution in original post

2 REPLIES 2

Re: Conda Environment in Nifi

The short answer is that you cannot embed such an environment to work within NiFi, but you can use NiFi to make calls against it.
NiFi embeds Jython, not full Python, so you cannot realistically get it to embed a full Anaconda environment. Typically what I do is use the ExecuteStreamCommand to call a small bash script which activates the python virtualenv and runs the script against the data, then returns to NiFi to carry on the flow.
Give this a try and let me know how you go.

View solution in original post

Highlighted

Re: Conda Environment in Nifi

@Dan Chaffelson,

I created a simple sh file as place the conda activate condaenv/source activate condaenv command and triggered it in command prompt and is working as expected(env is getting activated), however when I triggered the same sh file from Nifi(ExecuteStreamCommand), I'm getting error like conda not found. Not sure what's happening and why the same sh file working in prompt and not in Nifi. Any suggestions please.

Don't have an account?
Coming from Hortonworks? Activate your account here