Dear Matt, Thank you for your response; it was incredibly helpful. Based on your feedback and the link you shared, I have revised my design. Below are the key points regarding my implementation: All my data follows the same schema. Before merging, I convert the data to a JSON format to ensure consistency. There are no failures occurring during the merge process. The merge operation successfully produces a merged file. However, I am still receiving the original output, which I need to eliminate. Following the documentation, I have adjusted my design to include multiple consecutive merge steps, as illustrated in the attached diagram. Given the large volume of flow data, I have allocated high system specifications to optimise performance. Additionally, my heap size is currently set to 42 GB. Would increasing it further be beneficial? To mitigate swap-related issues, I have adjusted the swap space to 300,000 (from 20,000) since I previously experienced rapid memory fill-up. Could you advise on any potential side effects of this change? Looking forward to your insights.
... View more
