Support Questions

Find answers, ask questions, and share your expertise
Announcements
Now Live: Explore expert insights and technical deep dives on the new Cloudera Community BlogsRead the Announcement

Fork Record Processor cannot handle Large files

avatar
Contributor

I'm using ForkRecord processor in Apache NiFi for a use case where I need to retain the parent of the record and it is working smoothly for file sizes up to 100 MB. When I pass larger files say for example files with 250 MB or above it is taking longer time resulting in huge backlogs of files. 

Is there any alternate flow I can build to avoid such backlogs?

Can I completely avoid fork record processor and choose some other flow logic to solve this ?

 

@mburgess @MattWho   Please help here

1 REPLY 1

avatar
Super Guru

Can you break the data into smaller chunks ? this way you speed up the process by taking advantage of parallelism and multi threading.