I am having trouble with my dataflow using Nifi, Sure because I didn't understand how Wit/Notify Works. Here is my basic data flow:
- I listed my s3 bucket containing zip files.
- Extracted my Json file from the zip. Each Json is 2Gb up to 3 Gb.
- I then split my json several times to avoid 'Out of memory' to obtain sing records FlowFiles(Here comes the trouble).
- I want to flatten each flowfile
- I want to merge my flattened FF to re obtain the original Json. I know I cannot do with 'Merge Content' as my json goes through several split processors.
Can you please explain how to use the Wait/Notify?
Does the wait automatically merges the flowfiles?
Because I did not understand. I also looked at this post but it is still not clear.
Thanks