NiFi : Getting error in ExecuteStreamCommand while executing the Python Script

[#1]

I am getting below errors while trying to trigger a python script using ExecuteStreamCommand Processor in NiFi. Attaching the screenshot of the properties of the processor. Please help

Below is the error in bulletin board

23:21:24 CST

ERROR

3bc05a73-0170-1000-6f23-80001cc130eb

ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application

23:21:24 CST

ERROR

3bc05a73-0170-1000-6f23-80001cc130eb

ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Could not create external process to run command: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application

23:21:24 CST

ERROR

3bc05a73-0170-1000-6f23-80001cc130eb

ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application

23:21:24 CST

ERROR

3bc05a73-0170-1000-6f23-80001cc130eb

ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Could not create external process to run command: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application

23:21:24 CST

ERROR

3bc05a73-0170-1000-6f23-80001cc130eb

ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application

[#2]

@vikrant_kumar24 

If you open a terminal window on your Windows server, can you execute your python script?
If you are not successful from command line, the ExecuteStreamCommand processor is not going to be successful as well.

Thanks,

Matt

[#3]

Hi Mat I can execute the script from spyder (Anaconda) and it is successful. From executestreamcommand processor its not even triggering the script it seems. Its a .py file in the folder. Can this be an issue?

[#4 · Accepted solution]

@vikrant_kumar24 

The ExecuteStreamCommand processor is not doing anything fancy here.  It is just trying to execute the command you enter.  Assuming the command was "myscript.py", The executeStreamCommand would be doing same thing as you would if you opened a terminal window on your NiFi node and clicked on "myscript.py" to run it.  Keep in mind that NiFi is performing this action as the user that owns the NiFi java process.  The ExecuteStreamCommand processor will pass the FlowFiles content (if there is any) to stdin.

Have you perhaps looked at the ExecuteScript processor instead?

Hope this helps,

Matt

[#5]

Hi Matt,

I am able to execute python script from command line. Executescript processor is not an option here because it is an experimental process. I am looking for something more robust which can handle heavy data and also I should be able to put the flow in production. Can you please see the settings in my flow and see if I am doing anything wrong here?

[#6]

@vikrant_kumar24 

The ExecuteScript processor has been around for over 6 years as part of Apache NiFi.  It has had many improvements and bug fixes over those years just like many other well used components.  I'd be reluctant from calling it "experimental" any longer regardless of what the embedded Apache NiFi docs say.

The only thing to note here is that the ExecuteScript processor does not really execute the "Python" script engine.  It is executing "Jython" instead which is a Java implementation of Python.  Jython is not 100% compatible with Python, so you must test you script thoroughly. 

Thanks,

Matt

[#7]

Hi Matt, 

I am able to trigger python script now but stuck on how to pass the output file to STDout so that it can come back to flowfile. Here is what I am trying. STDIN seems to be working fine but STDOUT is not working

f = sys.stdin
reader = csv.reader(f)

***********

##Python code for creating the output file and passing it to STDOUT

xmlfile = open("C:\\Output_from_python\\output_file.xml",'w')
xmlFile.write(doc.toprettyxml(indent = '\t'))
xmlFile.close()
sys.stdout = xmlFile

[#8]

@vikrant_kumar24 

You would not configure your python script to write to an XML file on disk

NiFi handles the FlowFile creation in the framework. Any data passed by your Python script to STDOUT will be populated into the content of the resulting flowfile passed to the output stream relationship of the ExecuteStreamCommand processor. Your script does not need to have any awareness of what. FlowFile is or how it is created.

So you simply have your python script send the XML content to STDOUT and NiFi will take care of putting that content in to the FlowFile that will be produced and routed to the "output.stream" relationship of the processor.  You can then use the updateAttribute processor the change the filename associated with that content.

Hope this helps,

Matt

[#9]

@vikrant_kumar24  How were you able to run python script in ExecuteStreamCommand? Because I'm also getting the same error.

[#10]

I am having the same problem with head as the command, -1 as the argument, and csv in the content of my flowfile. I read someplace that nifi "lost the state"; what does that mean? I have the flowfile, it has content. Why can't the ESC processor send the first line of the file to the attribute I name?