Redirect Error Stream missing in ExecuteStreamCommand

[#1]

Hi,

Just started exploring Nifi to automate some of our data analytics flows. I am executing a spark-submit command from shell script using ExecuteStreamCommand processor and need to read the output from the shell script to check if it is success of failure. I think the output is getting redirected to STDERR as a result of which I can see the truncated output in the attribute 'execution.error'. The data is getting truncated as spark output can be huge. In 'ExecuteProcessor' there is a property 'Redirect Error Stream' which is putting all the output in flow file and works for me but I can connect upstream connections to Execute Processor. Planning to send the output to a file from shell script and read the status from there or see if I can automate extracting spark application id from nifi attribute and get logs from yarn, checking if there is any better solution which I may be missing.

Thanks

[#2]

Hi,

I am facing the same issue. Did you get any solution for your problem?

[#3]

I was also facing a similar kind of issue this code helped me!

Try this link:

https://github.com/mnemonic-no/act/blob/master/example-config/scio-act-workflow-2019-11-22.xml

It might help you!

Regards,

Lewis