Support Questions
Find answers, ask questions, and share your expertise

Is it possible for ExecuteStreamCommand to run a long running process?

I have a very high incoming event rate and a parser written in Python. I'm able to run it with ExecuteStreamCommand, but the latency for starting/stopping a process for every incoming FlowFile is too high and I cant' keep up with my source.

Is it possible for ExecuteStreamCommand to keep the external process alive and pass new FlowFiles after some signal?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Is it possible for ExecuteStreamCommand to run a long running process?

Contributor

There is not such a mechanism in place for ExecuteStreamCommand. ExecuteProcess does have the ability to batch but based on your mention of incoming event rate, seems like the need for handling input which ExecuteProcess does not provide.

Depending on the nature of your parsing/process, it may be possible to convert this to use InvokeScriptedProcessor which could tie said parser/process to the component lifecycle.

View solution in original post

2 REPLIES 2

Re: Is it possible for ExecuteStreamCommand to run a long running process?

Hi @Randy Gelhausen,

Is that parser written in Python using any specific module which is not supported by Jython? If parser is completely compatible with Jython, then "ExecuteScript" processor would work without overhead of spawning a new process for every incoming flowfile.

Re: Is it possible for ExecuteStreamCommand to run a long running process?

Contributor

There is not such a mechanism in place for ExecuteStreamCommand. ExecuteProcess does have the ability to batch but based on your mention of incoming event rate, seems like the need for handling input which ExecuteProcess does not provide.

Depending on the nature of your parsing/process, it may be possible to convert this to use InvokeScriptedProcessor which could tie said parser/process to the component lifecycle.

View solution in original post