Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi ETL: Principles or decision points for ETL on Nifi or more traditional tools (or pig batch)

avatar
Guru

Processors like ExecuteScript, ReplaceText and TransformXML provide powerful means to transform content and thus enable Nifi as an ETL tool. What principles/guidelines/decision points are there to evaluate when Nifi is not the best tool for ETL and when it is? Concerns include capabilities, performance, auditing, and so on.

1 ACCEPTED SOLUTION

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
2 REPLIES 2

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar

"ETL" which is embarassingly parallel (all processing logic can execute completely based purely on the contents of the incoming record itself) is in NiFi's sweet spot.

ETL which requires lookups for billions of records, or which must perform "group by" operations fits better in traditional Hadoop solutions like Hive, Pig, or Spark.