question Re: Performance diff between single big file vs multiple smaller files in Support Questions

Performance diff between single big file vs multiple smaller files

manishg — Wed, 27 Sep 2023 09:36:15 GMT

I have a nifi flow. I observe that if input file is split into smaller files and fed into the flow one by one, then overall time taken(sum of time taken for individual files) is considerably low compared to when I feed single big file.

What can be a possible cause for this performance difference?

Note:

Flow has many processors that use avro readers/writers.

I calculate time using following in a LogMessage processor:

${now():toNumber():minus(${lineageStartDate}):format("HH:mm:‌ss", "GMT")}

Re: Performance diff between single big file vs multiple smaller files

MattWho — Wed, 27 Sep 2023 12:54:32 GMT

@manishg

The first thing that comes to mind is JVM heap. You may want to collect and look at garbage collection data with large files versus small files.

Second would be identifying which processor(s) the largest FlowFiles spend the most time at. For this I would suggest looking at the provenance lineage for the large FlowFiles. There is a slide bar at bottom of that lineage graph that you can scroll to see progression of FlowFile through the lineage tree. What processor follow did the FlowFile spend longest at and how are they configured?

Hope this helps,

Matt

Re: Performance diff between single big file vs multiple smaller files

manishg — Thu, 28 Sep 2023 02:52:38 GMT

Where is this graph available on ui? And does it get updated after every run?

Re: Performance diff between single big file vs multiple smaller files

manishg — Thu, 28 Sep 2023 02:57:16 GMT

Got it. Its on Data Provenance dialog box.