Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Different frameworks to automate Hadoop jobs

Hello

What are the different frameworks to automate Hadoop jobs? Which is the best?

Thanks

2 REPLIES 2

Super Collaborator

Your open source options include

Oozie (included with HDP), Luigi, Airflow, and Azkaban.

https://www.bizety.com/2017/06/05/open-source-data-pipeline-luigi-vs-azkaban-vs-oozie-vs-airflow/

NiFi is also capable of scheduling tasks.

An enterprise option includes Stonebranch UAC

"Best" is relative to your needs and who will support it

New Contributor