question Re: Apache NiFi custom processor runtime environment error in Support Questions

Apache NiFi custom processor runtime environment error

cyrine — Wed, 24 Jul 2024 09:33:28 GMT

I am using NiFi 2.0 that supports python scripts directly from a working directory, after I add my custom processor to the canvas, an error "Processor invalid because initializing runtime environment for the processor" is indicated in each custom processor. I have already installed Java JDK 21, python 3.11 and all the requirements specified in the official documentation. How can I resolve this?

I have already installed Java JDK 21, python 3.11 and all the requirements specified in the official documentation.I changed the python extension parameters in the nifi.properties file.

This is what appears in the app-log:
2024-07-24 10:31:06,608 ERROR [Initialize CreateFlowFile] org.apache.nifi.NiFi An Unknown Error Occurred in Thread VirtualThread[#138,Initialize CreateFlowFile]/runnable@ForkJoinPool-1-worker-2
java.lang.RuntimeException: Failed to launch Process for Python Processor [CreateFlowFile] Version [0.0.1-SNAPSHOT]
at org.apache.nifi.py4j.StandardPythonBridge.getProcessForNextComponent(StandardPythonBridge.java:267)
at org.apache.nifi.py4j.StandardPythonBridge.createProcessorBridge(StandardPythonBridge.java:125)
at org.apache.nifi.py4j.StandardPythonBridge.lambda$createProcessor$7(StandardPythonBridge.java:142)
at org.apache.nifi.python.processor.PythonProcessorProxy.lambda$new$0(PythonProcessorProxy.java:78)
at java.base/java.lang.VirtualThread.run(VirtualThread.java:309)
Caused by: java.io.IOException: Failed to create Python Environment ./work/python/extensions/CreateFlowFile/0.0.1-SNAPSHOT: process existed with code 1
at org.apache.nifi.py4j.PythonProcess.setupEnvironment(PythonProcess.java:359)
at org.apache.nifi.py4j.PythonProcess.start(PythonProcess.java:129)
at org.apache.nifi.py4j.StandardPythonBridge.getProcessForNextComponent(StandardPythonBridge.java:248)
... 4 common frames omitted

Re: Apache NiFi custom processor runtime environment error

DianaTorres — Wed, 24 Jul 2024 18:38:16 GMT

@cyrine Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @joseomjr @MattWho who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Apache NiFi custom processor runtime environment error

Thressian — Mon, 12 Aug 2024 12:44:47 GMT

Hey,

I have been running into similar issues in recent weeks building custom processors, I have yet to find a satisfactory solution to this issue outside of removing the processors from the python/extensions folder, restarting, and then adding them back one by one.

Sometimes this will clear the issue, sometimes it won't.

Is this a known issue with Nifi 2.0, and is there any workaround or fix that the team is aware of?

Thanks.

Re: Apache NiFi custom processor runtime environment error

SAMSAL — Mon, 12 Aug 2024 15:06:36 GMT

Hi,

To help troubleshoot :

1- what version of nifi 2.0 are you using exactly and what system?

2- Can you post the error as appears in the log?

3- Do you see any dependency packages under the path /work/python/extension/{processor name}/2.0.0-SNAPSHOT. If the answer is no, can you provide screenshot of what us under that folder.

4- Do you see any file called "env-creation-complete.txt" under the same folder above?

5- Are you getting any error when starting nifi after enabling the python extension by uncommenting the following line :

nifi.python.command=...

Thanks

Re: Apache NiFi custom processor runtime environment error

rubendsm — Fri, 04 Oct 2024 14:59:31 GMT

i had the same issue
i made sudo apt install python3-venv
restarted apache nifi
and it worked