The first solution works! Thanks @Matt Burgess Here's the RouteOnAttribute configurations. ... View more

@Wynner It works with executeProcess but I don't want to use this processor. I use ExecuteStreamCommand so that I can route on attribute to detect failure. unfortunately the job gets stuck with it. ... View more

@Wynner please check my last comment, I figured more details of the problem, but it still exists. ... View more

@Wynner I figured out what is the problem, the spark job gets stuck when using "ExecuteStreamCommand" at some task, When I run the same command from "ExecuteProcess" or from the shell myself the job terminates successfully. I don't know what is the problem with "ExecuteStreamCommand" https://community.hortonworks.com/questions/135430/spark-job-gets-stuck-when-submitted-from-nifi.html?childToView=135436#answer-135436 ... View more

@Wynner I increased the concurrent tasks up to 3. but still having the same problem in different manner. If 3 tasks completed successfully it'll stuck. It seems like 'Spark-submit' command isn't terminated, even if it's terminated and its output is present in the file system. Do you know why this could happen? ... View more

The command is spark-submit <path to jar> ... View more

@Wynner It's a spark job, it takes approx. 1 minute and it terminates when I run it outside NiFi. What's the problem ? ... View more

@Wynner Here's what I'm trying to illustrate: one successful execution at "ExecuteStreamCommand" then it gets stuck (flow files keeps generated but ExecuteStream is stuck): ------ If no successful executions happens at all (All executions failed) the schedule works well as follows (flow files generated every minute, and executeStreamCommand executes every minute): I don't know why it gets stuck in the first case ? please help. ... View more

@Wynner Ok the schedule seems to be working, when the submitted job fails it works fine and the flow is ok. once the job run without errors, flow files keeps generated every minute, but the ExecuteStreamCommand is stuck. I can't even stop or start it, I need to restart NiFi to run it again. When I try to stop/start ExecuteStreamCommand it says: "No eligible components are selected. Please select the components to be stopped." ... View more

I don't understand exactly what do you mean. In a previous question the answer was : "You could schedule a GenerateFlowFile at the same rate your ExecuteProcess was scheduled for, and set Ignore STDIN to true in ExecuteStreamCommand. Then the outgoing flow files will have the execution.status attribute set, which you can use with RouteOnAttribute to handle failures (non-zero exit codes, e.g.)" I want to know how to schedule these 2 processors together so that the result is that the flow is executed every 1 minute. ... View more