Is it possible for ExecuteStreamCommand to run a long running process?

[#1]

I have a very high incoming event rate and a parser written in Python. I'm able to run it with ExecuteStreamCommand, but the latency for starting/stopping a process for every incoming FlowFile is too high and I cant' keep up with my source.

Is it possible for ExecuteStreamCommand to keep the external process alive and pass new FlowFiles after some signal?

[#2]

Hi @Randy Gelhausen,

Is that parser written in Python using any specific module which is not supported by Jython? If parser is completely compatible with Jython, then "ExecuteScript" processor would work without overhead of spawning a new process for every incoming flowfile.

[#3 · Accepted solution]

There is not such a mechanism in place for ExecuteStreamCommand. ExecuteProcess does have the ability to batch but based on your mention of incoming event rate, seems like the need for handling input which ExecuteProcess does not provide.

Depending on the nature of your parsing/process, it may be possible to convert this to use InvokeScriptedProcessor which could tie said parser/process to the component lifecycle.