Hello everyone,
I am facing issue with creating an custom python processor in Nifi 2.0.0. It is linux based server with installed java 22 and python 3.9.18. I also set the nifi.properties command to the python3.9.
Currently I am just trying to get the simplest processor such as
from nifiapi.flowfiletransform import FlowFileTransform, FlowFileTransformResult
class WriteHelloWorld(FlowFileTransform):
class Java:
implements = ['org.apache.nifi.python.processor.FlowFileTransform']
class ProcessorDetails:
version = '0.0.1-SNAPSHOT'
def __init__(self, **kwargs):
pass
def transform(self, context, flowfile):
return FlowFileTransformResult(relationship = "success", contents = "Hello World", attributes = {"greeting": "hello"})This is picked up by nifi and it seems that successfuly but in the GUI the process still has the warning 'Processor' is invalid because initializing runtime environment for the Processor.
From the logs I saw that there is mentioned /work/python/controller folder which is totally missing for me. When I change the __init__ to pass and manually copy the python interpreter to the /work/python/controller then the warning disappear but I get runtime error which can be seen in the log below
I also verified that under the /work/python/extensions/WriteHelloWorld/0.0.1-SNAPSHOT I see correct files with dependency-download.complete with content True and empty env-creation-complete.txt
In bin also I see all venv activate files created.
I also can manually enter the created venv using the activate.csh file. So the main problem seems to be with the communication between nifi and python. Nifi runs as admin.
Any idea how to solve it or find the rootcause of this problem? I already did all the steps from Apache NiFi custom processor runtime environment e... - Cloudera Community - 390917
What gets created is ".\work\python\extensions\[CustomProcessor]\2.0.0-SNAPSHOT".
it should have all downloaded dependencies as specified in your processor code, then venv files and folder, in case of windows it creates Script folder where all the venv files:
I think in Unix it store those files under Bin folder but it should not matters.
where you seeing any error messages or warning initially before manually creating the folder? Also one thing you can try is use later python version than 3.9 , I know in my case 3.11 worked well for me.
This is the warning I get without any manual creating folder. Meaning I just change the python.command to python add the .py and start nifi.
I upgraded the python to the python3.11 and set it as default but still same issue. Also maybe this can bring more information. This is the content of the pyvenv.cfg
Also I change the logs from python to get some outputs in nifi-app.log file maybe this gives you more information. It seems that the connection is there and the python communicate with java
So I took the cod you provided and created the python file and placed it unde:
.\python\extensions
After that I launched nifi (Im still on 2.0.0 M4) , then I went and select the WriteHelloWorld Processor and initially without chaning or adding anything else I was getting the same warning messages you have posted in your last message, however once I added the upstream processor because it needs it , then I started to get different warning messages about not connecting success, failure & original relationships which indicates that the processor has loaded successfully. Once I terminated the failure & original relationships the warning disappeared and I can run the flow to generate the Hello World message:
The flowfile in the success rel has the following output:
The following files and directories have been generated under .\work\python\extensions:
\WriteHelloWorld\0.0.1-SNAPSHOTUnder 0.0.1-SNAPSHOT folder:
Try to create the upstream processor. I know it should not make sense but maybe it needs it to complete the setup.
Hope that helps but at least it shows that there is nothing wrong with your code and its just matter of setup and configuration against your environment.
