question Re: should we need to restrart nifi everytime when we develop custom processor in Archives of Support Questions (Read Only)

should we need to restrart nifi everytime when we develop custom processor

siva_karna — Thu, 13 Jul 2017 20:27:25 GMT

Should we need to restart nifi cluster everytime when we develop a new custom processor?

Suppose if i have a nifi cluster or single node nifi ,one flowfile process group is processing the data continuosly(realtime data)

Later i want to design a new data flow which contains some custom processors , for that we should put nar files into the lib folder then we should restart the nifi again ,

so there is an abstraction for the first process group flow file it will stop so we will loss the data .

Is there any way to avoid this,any automation is there ?How we need to configure ,can you please describe me?

Re: should we need to restrart nifi everytime when we develop custom processor

MattWho — Thu, 13 Jul 2017 20:57:17 GMT

@siva karna

I am not following the statement "so there is an abstraction for the first process group flow file it will stop so we will loss the data". Why would stopping a dataflow cause data loss?

NiFi will only read in new nars/jars added to a NiFi lib directory on startup. There is no option to dynamically add classes during runtime.

Thanks,

Matt

Re: should we need to restrart nifi everytime when we develop custom processor

MattWho — Thu, 13 Jul 2017 21:10:47 GMT

NiFi stores FlowFile Attributes in the FlowFile repo and FlowFile Content in the Content repo. NiFi knows which queue FlowFiles were in when if it is shutdown. This allows Nifi to reload these FlowFiles back in to those queues and pick up where the dataflow left off after a restart.

Re: should we need to restrart nifi everytime when we develop custom processor

siva_karna — Thu, 13 Jul 2017 21:19:53 GMT

@Matt Clarke so we should restart nifi ,if we develop any processor , am i correct?

suppose while one dataflow processing data,later i want to add my custom processor to design another data flow.

then i shoould add that nar file to lib and i should restart nifi,

my doubt was , whatever the time period between stop and start nifi again, that time period data we could recieve or not?

Re: should we need to restrart nifi everytime when we develop custom processor

MattWho — Thu, 13 Jul 2017 21:34:34 GMT

@siva karna

Anytime you add, remove, or modify any nar in anyone of Nifi's lib directories, a restart will be needed. At startup NiFi extracts all those nars in to its work directory. To understand if you will lose data, you need to look at method/processors being used it ingest data in your NiFi. While NiFi is carefully in handling data it already has in its possession, it has not control over data that is being sent to it.

For example, any listen type processors will not be running so they will not be able to receive data while Nifi is restarting. Listen type processors that use TCP protocol should fail should trigger service unreachable/unavailable on sending side of connection. The sender should queue this data and continue to try and resend until service is available again. Now if you are using a listener that uses UDP protocol, that is a different story. There is no handshake there and you need to be willing to accept data loss by using that protocol for data transport.

In order to truly answer that question, you need to closely look at how your dataflow is designed to ingest data. NiFi takes care of not losing data once it is in its control as FlowFiles.

Thanks,

Matt

Re: should we need to restrart nifi everytime when we develop custom processor

siva_karna — Thu, 13 Jul 2017 22:13:19 GMT

Thank you for clarified me @Matt Clarke

Re: should we need to restrart nifi everytime when we develop custom processor

MattWho — Thu, 13 Jul 2017 22:17:13 GMT

@siva karna

Glad to help. If you found I addressed the question, please mark answer as accepted to close this thread.

Thanks,

Matt