Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi : Getting error in ExecuteStreamCommand while executing the Python Script

Solved Go to solution

NiFi : Getting error in ExecuteStreamCommand while executing the Python Script

I am getting below errors while trying to trigger a python script using ExecuteStreamCommand Processor in NiFi. Attaching the screenshot of the properties of the processor. Please help

 

Below is the error in bulletin boardBulletin Board.PNGExecuteStreamCommand.PNGProperties ExecuteStreamCommand.PNG

23:21:24 CST
ERROR
 
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
 
23:21:24 CST
ERROR
 
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Could not create external process to run command: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
 
23:21:24 CST
ERROR
 
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
 
23:21:24 CST
ERROR
 
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Could not create external process to run command: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
 
23:21:24 CST
ERROR
 
ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NiFi : Getting error in ExecuteStreamCommand while executing the Python Script

Master Guru

@vikrant_kumar24 

 

The ExecuteStreamCommand processor is not doing anything fancy here.  It is just trying to execute the command you enter.  Assuming the command was "myscript.py", The executeStreamCommand would be doing same thing as you would if you opened a terminal window on your NiFi node and clicked on "myscript.py" to run it.  Keep in mind that NiFi is performing this action as the user that owns the NiFi java process.  The ExecuteStreamCommand processor will pass the FlowFiles content (if there is any) to stdin.

 

Have you perhaps looked at the ExecuteScript processor instead?

 

Hope this helps,

Matt

View solution in original post

6 REPLIES 6
Highlighted

Re: NiFi : Getting error in ExecuteStreamCommand while executing the Python Script

Master Guru

@vikrant_kumar24 

 

If you open a terminal window on your Windows server, can you execute your python script?
If you are not successful from command line, the ExecuteStreamCommand processor is not going to be successful as well.

 

Thanks,

Matt

Highlighted

Re: NiFi : Getting error in ExecuteStreamCommand while executing the Python Script

Hi Mat I can execute the script from spyder (Anaconda) and it is successful. From executestreamcommand processor its not even triggering the script it seems. Its a .py file in the folder. Can this be an issue?

Highlighted

Re: NiFi : Getting error in ExecuteStreamCommand while executing the Python Script

Master Guru

@vikrant_kumar24 

 

The ExecuteStreamCommand processor is not doing anything fancy here.  It is just trying to execute the command you enter.  Assuming the command was "myscript.py", The executeStreamCommand would be doing same thing as you would if you opened a terminal window on your NiFi node and clicked on "myscript.py" to run it.  Keep in mind that NiFi is performing this action as the user that owns the NiFi java process.  The ExecuteStreamCommand processor will pass the FlowFiles content (if there is any) to stdin.

 

Have you perhaps looked at the ExecuteScript processor instead?

 

Hope this helps,

Matt

View solution in original post

Highlighted

Re: NiFi : Getting error in ExecuteStreamCommand while executing the Python Script

Hi Matt,

I am able to execute python script from command line. Executescript processor is not an option here because it is an experimental process. I am looking for something more robust which can handle heavy data and also I should be able to put the flow in production. Can you please see the settings in my flow and see if I am doing anything wrong here?

Highlighted

Re: NiFi : Getting error in ExecuteStreamCommand while executing the Python Script

Hi Matt, 

I am able to trigger python script now but stuck on how to pass the output file to STDout so that it can come back to flowfile. Here is what I am trying. STDIN seems to be working fine but STDOUT is not working

 

f = sys.stdin
reader = csv.reader(f)

***********

##Python code for creating the output file and passing it to STDOUT

xmlfile = open("C:\\Output_from_python\\output_file.xml",'w')
xmlFile.write(doc.toprettyxml(indent = '\t'))
xmlFile.close()
sys.stdout = xmlFile

Highlighted

Re: NiFi : Getting error in ExecuteStreamCommand while executing the Python Script

Master Guru

@vikrant_kumar24 

You would not configure your python script to write to an XML file on disk

 

NiFi handles the FlowFile creation in the framework. Any data passed by your Python script to STDOUT will be populated into the content of the resulting flowfile passed to the output stream relationship of the ExecuteStreamCommand processor. Your script does not need to have any awareness of what. FlowFile is or how it is created.

 

So you simply have your python script send the XML content to STDOUT and NiFi will take care of putting that content in to the FlowFile that will be produced and routed to the "output.stream" relationship of the processor.  You can then use the updateAttribute processor the change the filename associated with that content.

 

Hope this helps,

Matt

Don't have an account?
Coming from Hortonworks? Activate your account here