Support Questions
Find answers, ask questions, and share your expertise

How to replace ETL process with Nifi

Solved Go to solution
Highlighted

How to replace ETL process with Nifi

Dear all,


We will soon migrate to Big data platform along with Hortonworks. Yey !
Currently, we have large projects based on ETL/ELT processes using ODI and my company wants to migrate them, or at least the new projects to be done via Nifi. So, here is my question:
Knowing that Nifi is more like a data flow process, specifically for loading massive amount of data into Data lake for example, what are the features that Nifi make me able to replace an ETL/ELT process, precisely, concentrated on data transforming/checking part ? I know about SplitText, ExtractText and so on but I didn't see the specific filters...
For example, how can I check each record of a CSV/XLSX file if, i.e., has the correct length, to check if that record is really a number not a varchar, how to check if column X has a value then column Z should has also a value ? For the lines that their records are good, we insert in a repository table and then use it to make a join with a materialize view in order to provide a CSV file with specific content. Here is my second question:
How can we make such a join in Nifi and how to interact with the tables of a schema ? If you do not have time to answer, it will be great also a link which show how to solve this.
I think those questions are in mind of the all developers which want to migrate from ETL/ELT to Nifi.

Thank you for your time and help !

Stefan

1 ACCEPTED SOLUTION
2 REPLIES 2