question NiFi upgrade 1.9.2 to 1.22.0 and ListFile lost tracking in Support Questions

NiFi upgrade 1.9.2 to 1.22.0 and ListFile lost tracking

Soli — Tue, 05 Dec 2023 06:05:38 GMT

Hi , All

After we upgarding NiFi 1.9.2 to 1.22.0 (using cluster) , we found that ListFile will lost tracking. Just like @TimothySpann say.
see: https://community.cloudera.com/t5/Support-Questions/NiFi-upgrade-1-9-2-to-1-23-0/m-p/376353#M242873

Here is our ListFile setting :

Listing Strategy Tracking Entities

Recurse Subdirectories true

Record Writer No value set

Input Directory Location Remote

File Filter [^\.].*

Path Filter No value set

Include File Attributes true

Minimum File Age 1 min

Maximum File Age No value set

Minimum File Size 0 B

Maximum File Size No value set

Ignore Hidden Files true

Target System Timestamp Precision Auto Detect

Entity Tracking State Cache RedisDistributedMapCacheClientService

Entity Tracking Time Window 365 days

Entity Tracking Initial Listing Target All Available

Entity Tracking Node Identifier ${hostname()}

Track Performance false

Maximum Number of Files to Track 100000

Max Disk Operation Time 10 secs

Max Directory Listing Time 3 mins

We want to know thet reason why upgrade will cause lost tracking and duplicate files.
Is that isListingResetNecessary(see below) be triggered that cause duplicate files ?
https://github.com/apache/nifi/blob/rel/nifi-1.22.0/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFile.java

There are state save in redis　－＞ListedEntities::44793129-057f-1c10-af50-b96588dde6d7.
But it seems to not using this k=v in new NiFi

Re: NiFi upgrade 1.9.2 to 1.22.0 and ListFile lost tracking

TimothySpann — Wed, 06 Dec 2023 14:56:33 GMT

Did you try @MattWho https://community.cloudera.com/t5/Support-Questions/NiFi-upgrade-1-9-2-to-1-23-0/m-p/376353#M242873 solution. I think best course is to always start fresh after updates and restarts to ensure nothing is lost.

Re: NiFi upgrade 1.9.2 to 1.22.0 and ListFile lost tracking

Soli — Mon, 11 Dec 2023 08:40:11 GMT

Hi, @TimothySpann
Yes, our final upgrade version is 1.22.0 and there is not ghosted processors.
We want to know the basic principle of ListFile+Redis cache mapping. We found file state had some change that stored in "redis". The change is reflected in the precision of the file's modify time.
ps. We upgrade nifi on the same machine, time is still the same.

1.9.2
{'/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04594': {'timestamp': 1701853172000, 'size': 0}, '/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04593': {'timestamp': 1701853171000, 'size': 0}}
1.22.0
{'/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04594': {'timestamp': 1701853172307, 'size': 0}, '/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04593': {'timestamp': 1701853171506, 'size': 0}}