Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

Fork Record Processor cannot handle Large files

avatar
Contributor

I'm using ForkRecord processor in Apache NiFi for a use case where I need to retain the parent of the record and it is working smoothly for file sizes up to 100 MB. When I pass larger files say for example files with 250 MB or above it is taking longer time resulting in huge backlogs of files. 

Is there any alternate flow I can build to avoid such backlogs?

Can I completely avoid fork record processor and choose some other flow logic to solve this ?

 

@mburgess @MattWho   Please help here

1 REPLY 1

avatar
Super Guru

Can you break the data into smaller chunks ? this way you speed up the process by taking advantage of parallelism and multi threading.