How would you compare Nifi with Jenkins in-terms of Scheduling

[#1]

Hi all,

We have been using Jenkins for scheduling jobs, hence it is easy to schedule a job (or jobs, define dependencies etc) and keep track of each job run i.e if a job fails you get an alert etc. Hence for Operation teams, Jenkins is an easy platform to manage/keep track of jobs.

I have following questions:

1. For scheduling jobs, what is the best tool: Jenkins or Nifi?

2. How could you operationalize a dataflow like you can in Jenkins? Meaning, if any individual dataflow fails, Operation team gets an alert, so they have complete visibility on each job run?

3. Can I (or should I, meaning does it sound reasonable) use Jenkins to launch DataFlows on Nifi? Just to let Operation team have a single UI to keep track of all jobs !

4. How can we track the status of each DataFlow run on Nifi?

Thanks

Obaid

[#2 · Accepted solution]

I will try to answer your questions at a high level, and I (or others) can add detail as you need.

1. Jenkins and NiFi can both use CRON scheduling to start processing.

2. For any failure of a processor in a NiFi flow, you can route to a processor such as PutEmail to notify operation team members.

3. NiFi offers Data Provenance through which you can track how data flows through NiFi. Data Provenance provides a way to search the Data Provenance repositories for flowfile names, component IDs, and other attributes to determine what happened during the data flow. Jenkins has a more specific status display for jobs, but NiFi's Data Provenance UI may also serve your purpose, though it is more oriented towards data flow.

4. Data Provenance, described above, allows you to track the status of data flow in NiFi.

Do these answers help you, and are they detailed enough?

[#3]

Nifi's provenance UI provides very low level information. If operations team is interested in high level information then other option is publish provenance information to ElasticSearch and create a Kibana dashboard on top of it customized for operations team.