Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

NiFI Create multiple jobs

New Contributor

Hi Guys,

We are having multiple jobs in Informatica;

Job1: X application files load to Y DB

Job2: A application files load to Y DB

Job3: Y DB table load to Z DB

I want to create 3 jobs for 3 different needs; how do i achieve it using NIFI?




Expert Contributor

It sounds like you are trying to convert a job-based batch use case into a real-time streaming use case. You will have to re-think what your requirements and goals are for your process, as NiFi doesn't operate in jobs but in streaming data that is constantly coming in.

In NiFi, you may decide to implement your workflow the following way. Set your processor to monitor an HDFS directory for new files and load them into "Y DB" whenever new files are found with a scheduling interval of 1 minute (or 5, or 10). You would use the ListFiles processor to monitor for new files in a directory. That would accomplish your first 2 jobs. For the third job, you could use the QueryDatabaseTable to scan for changes every so often (say, 1 minute) and load those changes into "Z DB". This would move your processing from a batch-oriented nature to a real-time streaming workflow.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.