How to batch up FlowFiles into ORC?

[#1]

In my NiFi project I get messages one at a time. I convert these into AVRO and, as one followup step, I want to bin them up into ORC files for use with Hive. To that end, I would like to use a MergeContent processor. Is that a good approach? How to find the correct configuration?

[#2]

@Hellmar Becker

What format are your original messages in before you convert them to Avro?

[#3]

Original format is CSV.

[#4 · Accepted solution]

@Hellmar Becker

You may want to try merging your CSV files before sending them to the ConvertCSVtoAvro processor. Try configuring your MergeContent as follows:

Feed merged output from this processor to the ConvertCSVtoAvro, then that output to ConvertAvroToORC processor, and finally off to hive.

Thanks,

Matt

[#5]

Thanks, this works great! Still I wonder: can I force NiFi to create larger batches? I noticed that whenever the batch size exceeds the queue size, things get stuck.