Member since
03-03-2020
4
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
994 | 03-03-2020 10:28 PM |
03-03-2020
10:28 PM
This problem should be solved in NiFi 1.8.0. I tested the scenario in MiNiFi 0.5.0 (based on NiFi 1.7.0). After I executed find . -exec touch -m {} + ListFile would list files again. Ref:https://stackoverflow.com/questions/51088169/nifi-listfile-processor-is-not-detecting-file-changes-sometimes
... View more
03-03-2020
10:19 PM
1. ListS3:List all object file path in the specific bucket. 2. RouteOnAttribute:Filter out unused file (optional). 3. FetchS3Object :Fetch file. 4. UpdateAttribute:Rename filename(file path in bucket) to the specific path. ex: 5. PutS3Object :Put file to the specific bucket. PS:Once execute FetchS3Object, files will be load in memory. So, it's better to limit Back Pressure Object Threshold or Size Threshold in the Connection after FetchS3Object.
... View more