Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Dynamic Creation of Processors in NiFi

avatar
Expert Contributor

We have a system composed of many databases and tables and we want to use NiFi to query these tables based on our requirements. Since NiFi's QueryDatabaseTable processor is statically linked to a single table, what we intend to do is to dynamically generate many processors of this kind to match the number of our tables in our different systems. Is this possible using ExecuteScript processor (or anything similar)?

1 ACCEPTED SOLUTION

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
3 REPLIES 3

avatar

Hi @J. D. Bacolod please take a look at this HCC article for using the API to configure processors on the fly:

https://community.hortonworks.com/articles/3160/update-nifi-flow-on-the-fly-via-api.html

Hope that helps!

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Master Guru

Also, as of NiFi 1.3.0 / HDF 3.0.0, GenerateTableFetch accepts incoming connections/flow files, so you can use ListDatabaseTables -> GenerateTableFetch -> RPG -> Input Port -> ExecuteSQL to fully distribute the fetching of batches of rows across your NiFi cluster. The RPG -> Input Port part is optional and only used on a cluster if you want to fetch rows in parallel.