Member since
10-18-2024
12
Posts
3
Kudos Received
0
Solutions
05-20-2025
12:50 PM
@AndreyDE Glad I could help. Using just controlRate by itself has its limitation as well because it does not optimize for performance/throughput. You are setting some rate at which batches of FlowFiles will be passed downstream. Doing just this means multiple things can happen: 1. Rate is to short resulting in additional batches being passed downstream before previous batch completed processing. This could potentially lead to large backlogs in flows affecting downstream processing just as you were experiencing previously. 2. Rate is set to long resulting in downstream processing of a batch completing well before ControlRate releases next batch. Results in slower overall throughput. 3. If some exception occurs in downstream processing, nothing would prevent additional batches from being released into that downstream processing creating a huge backlog. The above are handled by the slightly more complex option C. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
11-25-2024
09:21 AM
Hi , I dont see a function toNumber in the record path syntax , so Im not sure how did you come up with this. It would be helpful next time if you provide the following information: 1- input format. 2- screenshot of the processor configuration causing the error. As for your problem , the easiest and more efficient way - than splitting records- I can think of is using the QueryRecrod processor. lets assume you have the following csv input: id,date_time
1234,2024-11-24 19:43:17
5678,2024-11-24 01:10:10 You can pass the input to the QueryRecord Processor with the following config: The query above is added as a dynamic property which will expose new relationship with the property name that you can use to get the desired output. The query syntax is the following: select id,TIMESTAMPADD(HOUR, -3,date_time) as date_time from flowfile The trick for this to work is how you configure the CSV Reader and Writer to set the expectation on how to parse datetime fields in the reader\writer services: For the CSVReader, Make sure to set the following: CSVRecordSetWriter: Output through Result relationship: id,date_time
1234,2024-11-24 16:43:17
5678,2024-11-23 22:10:10 Hope that helps. If it does, please accept solution. Thanks
... View more
11-20-2024
09:03 AM
1 Kudo
Hi, Im unable to replicate the error . Can you provide more details about your flow with the processor configurations. here is what I tried and it worked: GenerateFlowFile: EvaluateXPath: Output flowfile attributes:
... View more
11-19-2024
04:41 AM
1 Kudo
Yes, CSV Format is Custom Format
... View more
10-23-2024
09:55 AM
Thank you, the matter was in "/", indeed ))
... View more
10-20-2024
01:45 AM
@drewski7 wrote: @AndreyDE Is one flowfile going into the SplitText processor and outputting 10000 flowfiles? Yes - one flow file How big is the flowfile going into the SplitText processor? About 30 KB Or is the source of the pipeline recursively getting all objects in your S3 bucket? Yes, it searches all objects recursively
... View more