NiFi 2.12 – How to avoid a repeated DB lookup for every file

Hi all,

I'm looking for the recommended way to avoid making the same database call for every file in NiFi.

Environment

- Apache NiFi version: 2.12.0

- Deployment: [single node / cluster with X nodes]

Use case

Our filenames include a suffix, for example: SampleFileName_SampleSuffix.xml

For each file picked up by NiFi, we:

1. Read the suffix from the filename (SampleSuffix in the example above).

2. Look up that suffix in a database config table.

3. Use the result to decide where the file goes, for example move it to location XYZ or location ABC.

Problem

This DB lookup runs for every file. We process a large number of files, and many of them share the same suffix, so we keep making the same DB call and getting the same result.

Thanks in advance.