Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Redirect Error Stream missing in ExecuteStreamCommand

Highlighted

Redirect Error Stream missing in ExecuteStreamCommand

New Contributor

Hi,

Just started exploring Nifi to automate some of our data analytics flows. I am executing a spark-submit command from shell script using ExecuteStreamCommand processor and need to read the output from the shell script to check if it is success of failure. I think the output is getting redirected to STDERR as a result of which I can see the truncated output in the attribute 'execution.error'. The data is getting truncated as spark output can be huge. In 'ExecuteProcessor' there is a property 'Redirect Error Stream' which is putting all the output in flow file and works for me but I can connect upstream connections to Execute Processor. Planning to send the output to a file from shell script and read the status from there or see if I can automate extracting spark application id from nifi attribute and get logs from yarn, checking if there is any better solution which I may be missing.

 

Thanks

2 REPLIES 2

Re: Redirect Error Stream missing in ExecuteStreamCommand

New Contributor

Hi,

I am facing the same issue. Did you get any solution for your problem?

 

Re: Redirect Error Stream missing in ExecuteStreamCommand

New Contributor

I was also facing a similar kind of issue this code helped me!

Try this link:

https://github.com/mnemonic-no/act/blob/master/example-config/scio-act-workflow-2019-11-22.xml

It might help you!

Regards,

Lewis

Don't have an account?
Coming from Hortonworks? Activate your account here