Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi upgrade 1.9.2 to 1.22.0 and ListFile lost tracking

avatar
Explorer

Hi , All

After we upgarding NiFi 1.9.2 to 1.22.0 (using cluster) , we found that ListFile will lost tracking. Just like @TimothySpann  say.
see: https://community.cloudera.com/t5/Support-Questions/NiFi-upgrade-1-9-2-to-1-23-0/m-p/376353#M242873


Here is our ListFile setting :

Listing Strategy Tracking Entities
Recurse Subdirectories true
Record Writer No value set
Input Directory Location Remote
File Filter [^\.].*
Path Filter No value set
Include File Attributes true
Minimum File Age 1 min
Maximum File Age No value set
Minimum File Size 0 B
Maximum File Size No value set
Ignore Hidden Files true
Target System Timestamp Precision Auto Detect
Entity Tracking State Cache RedisDistributedMapCacheClientService
Entity Tracking Time Window 365 days
Entity Tracking Initial Listing Target All Available
Entity Tracking Node Identifier ${hostname()}
Track Performance false
Maximum Number of Files to Track 100000
Max Disk Operation Time 10 secs
Max Directory Listing Time 3 mins

 We want to know thet reason why upgrade will cause lost tracking and duplicate files.
Is that isListingResetNecessary(see below) be triggered that cause duplicate files ?
https://github.com/apache/nifi/blob/rel/nifi-1.22.0/nifi-nar-bundles/nifi-standard-bundle/nifi-stand...

There are state save in redis ->ListedEntities::44793129-057f-1c10-af50-b96588dde6d7.
But it seems to not using this k=v in new NiFi

2 REPLIES 2

avatar
Master Guru

Did you try @MattWho https://community.cloudera.com/t5/Support-Questions/NiFi-upgrade-1-9-2-to-1-23-0/m-p/376353#M242873 solution.   I think best course is to always start fresh after updates and restarts to ensure nothing is lost.

avatar
Explorer

Hi, @TimothySpann 
Yes, our final upgrade version is 1.22.0 and there is not  ghosted processors.
We want to know the  basic principle of ListFile+Redis cache mapping. We found file state had some change that stored in "redis". The change is reflected in the precision of the file's modify time.
ps. We upgrade nifi on the same machine, time is still the same. 

1.9.2
{'/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04594': {'timestamp': 1701853172000, 'size': 0}, '/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04593': {'timestamp': 1701853171000, 'size': 0}}
1.22.0
{'/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04594': {'timestamp': 1701853172307, 'size': 0}, '/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04593': {'timestamp': 1701853171506, 'size': 0}}