Support Questions
Find answers, ask questions, and share your expertise

Limiting generation of multiple flow files from an ExecuteSQL processor in NIFI to 1 flow file

I am using a jdbc driver to get connected through the MS SQL 2012 server instance with my Nifi 1.3.0 instance. If I use QueryDatabaseTable processor I am able to fetch the incremental data for that specific table with the output just as 1 flow file. I want to fetch a table using a custom query(which includes 1 query which fetches different columns and from multiple tables), so I am using ExecuteSQL processor to run the query but the output of that processor is multiple flow files. I only need 1 flow file so that I can write a code to fetch the same incrementally and would be easy to track 1 flow file. Any idea as to how can this be achieved?

1 REPLY 1

Super Guru

ExecuteSQL will run on a scheduled basis. QueryDatabaseTable also run on a scheduled basis, but will check to see if any "new" rows are present and will only output a flow file if there are results. ExecuteSQL can return empty flow files (for empty result sets), so it will be fully executed when scheduled. It sounds like you are getting the same contents over and over again? If you want a single flow file, then you probably want to only run ExecuteSQL once, so you can either schedule it over a very long interval (once per day, e.g.), or schedule for something like 10 seconds and after starting it, immediately stop it. It will be guaranteed to be scheduled and run once (even if it hasn't run by the time you "stop" the processor). This is a manual process on the UI but can also be achieved programmatically using the REST API.

; ;