question Re: How to batch up FlowFiles into ORC? in Archives of Support Questions (Read Only)

How to batch up FlowFiles into ORC?

Former Member — Tue, 21 Apr 2026 12:26:52 GMT

In my NiFi project I get messages one at a time. I convert these into AVRO and, as one followup step, I want to bin them up into ORC files for use with Hive. To that end, I would like to use a MergeContent processor. Is that a good approach? How to find the correct configuration?

Re: How to batch up FlowFiles into ORC?

MattWho — Fri, 18 Nov 2016 02:30:44 GMT

@Hellmar Becker

What format are your original messages in before you convert them to Avro?

Re: How to batch up FlowFiles into ORC?

Former Member — Fri, 18 Nov 2016 02:31:33 GMT

Original format is CSV.

Re: How to batch up FlowFiles into ORC?

MattWho — Mon, 19 Aug 2019 11:35:15 GMT

@Hellmar Becker

You may want to try merging your CSV files before sending them to the ConvertCSVtoAvro processor. Try configuring your MergeContent as follows:

Feed merged output from this processor to the ConvertCSVtoAvro, then that output to ConvertAvroToORC processor, and finally off to hive.

Thanks,

Matt

Re: How to batch up FlowFiles into ORC?

Former Member — Sat, 19 Nov 2016 02:55:54 GMT

Thanks, this works great! Still I wonder: can I force NiFi to create larger batches? I noticed that whenever the batch size exceeds the queue size, things get stuck.