Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

Who agreed with this solution

avatar
Master Mentor

@garoosy 

You should look in to using the "ExecuteSQLRecord" instead of "ExecuteSQL" for large volume data.  To be efficient here you  would have many records in a single FlowFile.

Right now you have a single record per each FlowFile which is not going to be very efficient.  The only way for "ExecuteSQL" to handle multiple FlowFile executions in a single connection is if the SQL statement used in every FlowFile is identical.  In order to do that the unique values would need to come from FlowFile attributes.

You may find these post helpful:
https://community.cloudera.com/t5/Support-Questions/Nifi-ExectueSQL-how-to-force-a-parameter-to-be-a...

https://stackoverflow.com/questions/63330790/using-nifi-executesqlrecord-with-parameterized-sql-stat...

 

If you have threads that never seem to complete (will see small number in upper right corner of processor (2)), it is best to get a series of thread dumps (4 - 6) to verify thread is not progressing.  Then you have to determine if what the thread is waiting on.  Did you try setting a "Max Wait Time" on the processor?  It defaults to 0 which means it would wait forever.

 

Hope this helps,

Matt

View solution in original post

Who agreed with this solution