Fork Record Processor cannot handle Large files

Jagapriyan — Wed, 09 Oct 2024 06:25:16 GMT

I'm using ForkRecord processor in Apache NiFi for a use case where I need to retain the parent of the record and it is working smoothly for file sizes up to 100 MB. When I pass larger files say for example files with 250 MB or above it is taking longer time resulting in huge backlogs of files.

Is there any alternate flow I can build to avoid such backlogs?

Can I completely avoid fork record processor and choose some other flow logic to solve this ?

@mburgess @MattWho Please help here

Re: Fork Record Processor cannot handle Large files

SAMSAL — Wed, 09 Oct 2024 15:37:45 GMT

Can you break the data into smaller chunks ? this way you speed up the process by taking advantage of parallelism and multi threading.

question Re: Fork Record Processor cannot handle Large files in Support Questions

Fork Record Processor cannot handle Large files

Re: Fork Record Processor cannot handle Large files