Hi
Initially I thought this was fixed in M3 and felt quite the relief for not having to manually rename the scripts folder to bin and restarting nifi twice for that: once to create the initial extension folders under the work folder, and second after renaming the scripts to bin. It turns out that it works for the very first time however upon subsequent restarts the python extension processors start going into invalid state. When I check the log I found that its still looking for the bin folder:
2024-06-12 15:58:22,105 ERROR [Initialize ExcelTableToJson] org.apache.nifi.NiFi An Unknown Error Occurred in Thread VirtualThread[#118,Initialize ExcelTableToJson]/runnable@ForkJoinPool-1-worker-2: java.lang.RuntimeException: Failed to launch Process for Python Processor [ExcelTableToJson] Version [2.0.0-SNAPSHOT]
java.lang.RuntimeException: Failed to launch Process for Python Processor [ExcelTableToJson] Version [2.0.0-SNAPSHOT]
at org.apache.nifi.py4j.StandardPythonBridge.getProcessForNextComponent(StandardPythonBridge.java:262)
at org.apache.nifi.py4j.StandardPythonBridge.createProcessorBridge(StandardPythonBridge.java:123)
at org.apache.nifi.py4j.StandardPythonBridge.lambda$createProcessor$7(StandardPythonBridge.java:140)
at org.apache.nifi.python.processor.PythonProcessorProxy.lambda$new$0(PythonProcessorProxy.java:78)
at java.base/java.lang.VirtualThread.run(VirtualThread.java:309)
Caused by: java.io.IOException: Cannot run program "F:\nifi-2.0.0-M3-test\.\work\python\extensions\ExcelTableToJson\2.0.0-SNAPSHOT\bin\python": CreateProcess error=2, The system cannot find the file specified
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1170)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1089)
at org.apache.nifi.py4j.PythonProcess.launchPythonProcess(PythonProcess.java:283)
at org.apache.nifi.py4j.PythonProcess.start(PythonProcess.java:129)
at org.apache.nifi.py4j.StandardPythonBridge.getProcessForNextComponent(StandardPythonBridge.java:243)
... 4 common frames omitted
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.base/java.lang.ProcessImpl.create(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:500)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:159)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1126)
... 8 common frames omitted
2024-06-12 15:58:22,109 ERROR [Initialize ExcelTableToJson] org.apache.nifi.NiFi An Unknown Error Occurred in Thread VirtualThread[#116,Initialize ExcelTableToJson]/runnable@ForkJoinPool-1-worker-7: java.lang.RuntimeException: Failed to launch Process for Python Processor [ExcelTableToJson] Version [2.0.0-SNAPSHOT]
java.lang.RuntimeException: Failed to launch Process for Python Processor [ExcelTableToJson] Version [2.0.0-SNAPSHOT]
at org.apache.nifi.py4j.StandardPythonBridge.getProcessForNextComponent(StandardPythonBridge.java:262)
at org.apache.nifi.py4j.StandardPythonBridge.createProcessorBridge(StandardPythonBridge.java:123)
at org.apache.nifi.py4j.StandardPythonBridge.lambda$createProcessor$7(StandardPythonBridge.java:140)
at org.apache.nifi.python.processor.PythonProcessorProxy.lambda$new$0(PythonProcessorProxy.java:78)
at java.base/java.lang.VirtualThread.run(VirtualThread.java:309)
Caused by: java.io.IOException: Cannot run program "F:\nifi-2.0.0-M3-test\.\work\python\extensions\ExcelTableToJson\2.0.0-SNAPSHOT\bin\python": CreateProcess error=2, The system cannot find the file specified
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1170)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1089)
at org.apache.nifi.py4j.PythonProcess.launchPythonProcess(PythonProcess.java:283)
at org.apache.nifi.py4j.PythonProcess.start(PythonProcess.java:129)
at org.apache.nifi.py4j.StandardPythonBridge.getProcessForNextComponent(StandardPythonBridge.java:243)
... 4 common frames omitted
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.base/java.lang.ProcessImpl.create(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:500)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:159)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1126)
... 8 common frames omitted
When rename the scripts folder to bin and restart Nifi again its start working again!
Also Im not sure why there is no separate python.log file anymore like in M1 & M2, instead everything related to python extension is getting logged into the main nifi-app.log which makes it harder to find.
I have already posted a comment in github under pull request 12514 which seem to be related to this issue.
@pvillard
***UPDATE***:
I think I know why this is happening and under what circumstance. I notice that whenever this happens actually the bin folder is getting created upon downloading the dependencies. For example if you look at the log above I'm creating python processor that reads an excel table and converts into array of json records. This processor uses the following dependencies:
['pandas','numpy','openpyxl']
When they get downloaded a bin folder is getting created with the following file insde:
f2py.exe
I think this confuses Nifi and once its find the bin folder it thinks thats where venv files are therefore the processor goes into invalid state when it cant find them there.
That is why I think its better not to hard code the folder path to look for one or the other and depending on what if finds first (which in this case the bin folder) it will start looking there. I think this is better if its left to be configurable depending on which evn Nifi is deployed on.
Thanks
