Support Questions

Find answers, ask questions, and share your expertise

Fork Record Processor cannot handle Large files

avatar
Contributor

I'm using ForkRecord processor in Apache NiFi for a use case where I need to retain the parent of the record and it is working smoothly for file sizes up to 100 MB. When I pass larger files say for example files with 250 MB or above it is taking longer time resulting in huge backlogs of files. 

Is there any alternate flow I can build to avoid such backlogs?

Can I completely avoid fork record processor and choose some other flow logic to solve this ?

 

@mburgess @MattWho   Please help here

1 REPLY 1

avatar
Super Guru

Can you break the data into smaller chunks ? this way you speed up the process by taking advantage of parallelism and multi threading.