I am trying to Join multiple tables using NiFi. The datasource may be MySQL or RedShift maybe something else in future. Currently, I am using ExecuteSQL processor for this but the output is in a Single flowfile. Hence, for terabyte of data, this may not be suitable. I have also tried using generateTableFetch but this doesn't have join option.
Here are my Questions:
Please share your thoughts. Thanks in advance
Nifi is not the best tool to do table joins. It will be able to do some transformations, but the heavy lifting should be done with spark or storm (if real time is needed).
Have a look at this thread: https://community.hortonworks.com/questions/64667/joining-tables-within-nifi.html