Member since
09-13-2023
4
Posts
0
Kudos Received
0
Solutions
12-11-2023
12:40 AM
Hi, @TimothySpann Yes, our final upgrade version is 1.22.0 and there is not ghosted processors. We want to know the basic principle of ListFile+Redis cache mapping. We found file state had some change that stored in "redis". The change is reflected in the precision of the file's modify time. ps. We upgrade nifi on the same machine, time is still the same. 1.9.2 {'/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04594': {'timestamp': 1701853172000, 'size': 0}, '/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04593': {'timestamp': 1701853171000, 'size': 0}} 1.22.0 {'/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04594': {'timestamp': 1701853172307, 'size': 0}, '/mnt/EDA_DATA/NIFI/cloud/upgrade_test/ControlRate5/04593': {'timestamp': 1701853171506, 'size': 0}}
... View more
12-04-2023
09:40 PM
Hi , All After we upgarding NiFi 1.9.2 to 1.22.0 (using cluster) , we found that ListFile will lost tracking. Just like @TimothySpann say. see: https://community.cloudera.com/t5/Support-Questions/NiFi-upgrade-1-9-2-to-1-23-0/m-p/376353#M242873 Here is our ListFile setting : Listing Strategy Tracking Entities Recurse Subdirectories true Record Writer No value set Input Directory Location Remote File Filter [^\.].* Path Filter No value set Include File Attributes true Minimum File Age 1 min Maximum File Age No value set Minimum File Size 0 B Maximum File Size No value set Ignore Hidden Files true Target System Timestamp Precision Auto Detect Entity Tracking State Cache RedisDistributedMapCacheClientService Entity Tracking Time Window 365 days Entity Tracking Initial Listing Target All Available Entity Tracking Node Identifier ${hostname()} Track Performance false Maximum Number of Files to Track 100000 Max Disk Operation Time 10 secs Max Directory Listing Time 3 mins We want to know thet reason why upgrade will cause lost tracking and duplicate files. Is that isListingResetNecessary(see below) be triggered that cause duplicate files ? https://github.com/apache/nifi/blob/rel/nifi-1.22.0/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFile.java There are state save in redis ->ListedEntities::44793129-057f-1c10-af50-b96588dde6d7. But it seems to not using this k=v in new NiFi
... View more
Labels:
- Labels:
-
Apache NiFi
09-13-2023
06:19 PM
Hi @TimothySpann ,thanks for reply. We use a lot of ListFile (tracking entity)+Redis cache map. We were worried about state will be losted and cause duplicate files. Will it happend? Also, is there any recommended upgrade steps to follow for many versions migration?
... View more
09-13-2023
12:33 AM
Hi All , I planning to upgrade NiFi 1.9.2 to 1.23.0. I read some articles about upgrading. https://community.cloudera.com/t5/Support-Questions/Nifi-Upgrade/m-p/346309 If I follow the take special note above , can I directly migrate 1.9.2 to 1.23.0 ? Will there be any problem ?
... View more
Labels:
- Labels:
-
Apache NiFi