Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to configure hundreds data pipelines easily

How to configure hundreds data pipelines easily

Explorer

We have pipelines to read hundreds of batch files over the NIFI to read, validate each file (its formats specific to each file) and then ingest into HDFS. But configuring each file, creating its own validation rules over NIFI GUI is pain. Can we use any open source tool or any other method to make this configuration easier? Should we use schema approach instead where the source supplies a schema with the file and schema Id in the file? NIFI will open up the file and validate it against its schema in a generic way? Is there any piece of code doing that? Thanks..