Member since
04-20-2023
6
Posts
0
Kudos Received
0
Solutions
12-04-2023
10:37 AM
I'm using NiFi to process .tar.gz files, extract the csv files, and upload them into S3. NiFi is currently running on a t2.large. I tested the same workflow on a t2.2xlarge, expecting an increase in performance. Yet, it didn't happen. In particular, it seems like the CompressContent and UnpackContent processors take the same amount of time, regardless of the instance type. Any idea of what I'm doing wrong or how to optimize performance? For the t2.large, I have concurrent tasks set to 2, while for the t2.2xlarge I have it set to 8. Execution on all nodes for both. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
05-11-2023
08:13 AM
@cotopaul Thanks for replying. Apologize for not explaining myself properly. Here is the situation. The inititial data in the directory s3.bucket/data/to/be/processed/ looks like the following: file0000.tar.gz file0015.tar.gz ... file2345.tar.gz where 0000-0015-2345 are timestamp in 15 minutes increments. The end result should look like the following (same directory s3.bucket/data/to/be/processed/): file0000/exctraded_files file0015/extracted_files ... file2345/extracted_files where the folder is named after the original file, and it contains the extracted files. What I am experiencing is that most of the folders are created as expected and the files are correctly extracted. However, I often get two or three files that seem not to get processed. I checked as you suggested if it might be a naming issue but that's not the case. I run another job pointing at those files who were extracted in the first place and the end result is what I expect. Hope this was clear. Thanks for the help.
... View more
05-10-2023
12:43 PM
Hi all, I have several jobs processing daily files. In short, I load zipped files from a S3 directory, extract the files, and reload them into the same directory into a specific folder created by NiFi. The workflow works great, but it looks like some files are not being processed. I checked NiFi for any errors or warnings but it seems like everything is working fine. Has anyone experience a similar problem? If so, how do you fixed it? Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
04-20-2023
10:40 AM
@cotopaul I'm using version 1.18.0, so I guess I'd need to update to a newer version.
... View more
04-20-2023
08:07 AM
From other topics, I saw you are NiFi experts and took the liberty to tag you into this. I apologize ahead if it is against the rules @MattWho @SAMSAL @steven-matison @DigitalPlumber @cotopaul . Thank you for your help!
... View more
04-20-2023
07:59 AM
I need to List and Fetch specific files on S3 from April 2022 to December 2022. I was hoping to narrow the listing using the Minimum and Maximum Object Age properties as specificed in the processor documentation. However, I can;t find the Maximum Object Age property? Anybody knows how to use it? Any other strategy that might be useful? Listing all the files in the prefix takes forever as it contains a huge amount of data. Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi