Hi, We have a process that generates text files in a ftp server. The process generates the files by writing multiple passes and could take from say more than a minute to a few minutes to generate the files.   The nifi ListSFTP in this case generates multiple files as we are using tracking timestamps. The file's timestamp is getting updated continuously during the generation process.   We dont have control over the existing process that generates the files and has no notification mechanism in place that can tell us when the file generation is complete.     What would be the best practice to handle this scenario in a List+FetchSFTP flow to avoid duplicate files generated in the target system.    Please advice.   Thank you ... View more

Hi, We have  a requirement to retrieve files from SFTP server and place into to different prefixes in a S3 bucket.   Mapping is something like this:   ftp path1 -> file pattern match1 -> s3 prefix1 ftp path1 -> file pattern match2 -> s3 prefix2 ftp path2 -> file pattern match3 -> s3 prefix3 ftp path3 -> match4 -> s3 prefix4   ListSFTP -> FetchSFTP -> RouteOnAttribute  RouteOnAttribute - Match1 (${filename:indexOf('pattern1'):gt(-1)}) - putS3Object1 RouteOnAttribute - Match2 (${filename:indexOf('pattern2'):gt(-1)}) - putS3Object2   We are using the ListSFTP, FetchSFTP and RouteOnAttribute to route it to the a corresponding putS3Object processor with different variables for different patterns.   We initially had only 2 patterns which grew into 10 - 15 patterns (Match1, Match2, .. Match N) to be routed to the corresponding paths. So if we follow the same approach we have to use 10 -15 putS3Object. Is there a way to avoid this?   Preferably is there a way to match patterns on the file names, to a lookup table to identify a prefix for the corresponding pattern?   Is there a pattern that we can use to achieve this?   Thank you ... View more