NiFi confirm before we stop a data flow

[#1]

Hi All,

Thanks a lot everyone on this awesome community.

I have data flows in PROD now, and I am adding new data flows and testing them, however accidentally I stopped one the flows which was good and running in PROD, reuslting in data loss. Is there any way I can freeze the data flow or dataflow confirms twice before I stop a data flow?

Thanks

Dheeru

[#2 · Accepted solution]

@dhieru singh

I am not clear on what you mean by "stopping a dataflow resulted in data loss"? NiFi does not delete any data when a dataflow is stopped. Data will remain queued between the stopped components until the dataflow is restarted or a user manual operation is performed to purge the data from those queues.

There is no notion of a "lock" in NIFi that can be set on a component or set of components in NiFi. In addition, having a a double confirm every time a user wants to stop a component to make an edit may be more annoying then beneficial.

That being said, it might be an interesting idea to add the ability to "lock" the current running state of "process group". Basically putting all components in a process group in to read only mode until the lock is removed. Might be worth you creating an NiFi Apache Jira for such a thing.

If you Nifi is secured, you can prevent such issues by taking away users "modify" access to the components. Without Modify access policies, users can only view the components. They will not be able to change active state (start, stop, enable, or disable) or the configuration. But this would also require that you re-add "modify" anytime a change is desired.

Thank you,

Matt

[#3]

@Matt Clarke Thanks a lot for your response. I clicked on a process group which was already running fine and good, while my intention was to just see its configuration I accidentally clicked on "Stop". So the whole process group stopped and then we did not receive the data from the pfsense firewall for that little duration.

what I was looking for exactly what you suggested, like when we know the data flow is ready and does not need any more changes, we can change it to read mode only or Stop, options freezes.

Yes I agree completely asking twice for confirmation will be annoying, but the dataflows which are ready and we mark them to ask us twice, that can be a good feature.

Thank you

Dheeru

[#4]

Matt,

In the evaluation I'm doing NiFi versus StreamSets as a user, I prefer NiFi due to its richest pipeline management and clustering option in opensource version, and a lot of other things. The question posted by dhieru singh make enormous sense, in a perspective that a lot of critical systems (or even not critical ones) do this. I mean, if you accidentally stop all process groups at once and then, as soon you noticed that, restart all process again (imagine a DevOps guy waking up at 3:00AM 🙂 ) you may also start process groups that you don't want. This would be the second accident.

It happened with me during evaluation, where I had built several process groups and some of them were intentionally suspended (ok, good practice says if you don't use, remove it, but, sure I'm not the only one when it comes to testing).

Anyway, maybe is a good practice/idea to notify the user when he/she is about to stop the whole thing or, as you mention, add the ability to "lock" the current running state of a given process group.

Thank you !

Julio

[#5]

We have had unintended starts and stops that have hampered our data flows. We have talked to Hortonworks about it.

Mike

,

I have had the same experience. We have asked HWX for a confirmation window for these type of operations...