question Nifi best parctice to filter flowfiles using external file in Archives of Support Questions (Read Only)

Nifi best parctice to filter flowfiles using external file

nahum_liran — Sun, 06 Nov 2016 13:08:37 GMT

Hi,

Just wondering what is the best practice for my use-case. My flowfiles are json objects and I need to filter/route them using external file (with list of values) - i.e. per flowfile to check if the value of some field (key) X is in the file or not.

The only two processors I noticed I can use for that are ScanContent and ReplaceTextWithMapping (which will "replace" a value in identical one).

ScanContent seems to be more appropriate since it does not perform a redundant 'Replace' action, but on the other hand it does not have the 'File Refresh Interval' property as the ReplaceTextWithMapping. Hence I'm guessing it continuously refresh the dictionary file (I didn't find relevant information about this issue in the documents), which is also an expensive (and redundant for my use-case) action that can harm the performance of the flow.

I tend to use the ReplaceTextWithMapping approach and skip the continuous refreshing of the file, but just wanted to ask around here, to check if there is another best-practice approach and make sure I get things right / didn't miss something.

Thanks in advance,

Liran

Re: Nifi best parctice to filter flowfiles using external file

mburgess — Sun, 06 Nov 2016 23:15:56 GMT

The code for ScanContent looks like it watches the specified file for changes; otherwise it shouldn't refresh the dictionary file (unless something weird happens with the internal search mechanism).

Alternatively, I answered a Stack Overflow question with a similar use case, using ExecuteScript to check the JSON (and in their case, replace the value from an external file). That example also reads the file every time, but you could use a similar approach with InvokeScriptedProcessor to read the file in the initialize() method, then it will not be re-read during onTrigger (which is called when the processor is scheduled).

Re: Nifi best parctice to filter flowfiles using external file

nahum_liran — Mon, 07 Nov 2016 12:55:18 GMT

if the ScanContent watch for changes, I think it solve my problem. Thanks ! 🙂