Hi, just starting off with NiFi. I am impressed with the orchestration capabilities and am now considering how to best handle the "T" in ETL. Ultimately I need to support hundreds of different formats of structured data, typically transforming into one or two target formats. Often these formats have hundreds of fields and significant munging is required to reach the target format. External enrichment might be required (NER, geocoding, you name it.)
My thought right now is that I'll need a NiFi processor that handles translations using some external capability. Either my team can build this definition-driven translation engine and make it available to NiFi via a processor.... OR, has anyone here ever successfully integrated some other ETL tool? Open source and even commercial seems to have extraction and loading covered but I've not yet run into a great tool that specializes in allowing a data guru to create deep schema transformation definitions that can be executed at run time. The idea is roughly like XSLT in raw spirit but more more extensive in transformation capabilities.... would love to hear your recommendations if you have any.