Hello, I have a MergeRecord processor that is not merging despite my conditions (with the exception of bin age).
I have configured:
- 1 minimum record
- 2000 maximum records
- 1 MB minimum size
- no size limit
- 10 bins
- 30 min max age
- bin packing strategy, no correlation attribute
I have a 3 node cluster, and the queue before the MergeRecord processor has got millions of files before it that reach a couple Gigabytes in size.
I can see the processor keeps opening/closing tasks (by the thread count icon on it), but no files are getting merged and outputted, except for the bin age being reached. I believe all the minimum merge requirements are met, and even the max record limit should be reached, yet the processor isn't working as I understand. I would appreciate any help debugging why it is not merging as expecteded.
If it is relevant, I use a json reader and parquet writer.
Thanks,
Eyal.
