@Andi Chirita If you are already experienced with Oozie and your work is in Hadoop then look at Apache Falcon. It is part of Hortonworks Data Platform as well. I really like a tool to do more than just scheduling. I want the tool to execute various tasks or delegate execution not only on time but also on event or when conditions are met. Both, Apache Falcon and Apache NiFi can help with that. They are not specialized schedulers, just more than that. Falcon can satisfy your requirements if you live in Hadoop. If you want to do more than that, then look at Apache NiFi. "Falcon simplifies the development and management of data processing pipelines with a higher layer of abstraction, taking the complex coding out of data processing applications by providing out-of-the-box data management services. This simplifies the configuration and orchestration of data motion, disaster recovery and data retention workflows. The Falcon framework can also leverage other HDP components, such as Pig, HDFS, and Oozie. Falcon enables this simplified management by providing a framework to define, deploy, and manage data pipelines". Check here for what Falcon does, how it works etc: http://hortonworks.com/apache/falcon If you want to know more about this project backed by Hortonworks, go to: https://falcon.apache.org/ Apache NiFi is a tool to build a dataflow pipeline (flow of data from edge devices to the datacenter). NiFi has a lot of inbuilt connectors (known as processors in NiFi world) so it can Get/Put data from/to HDFS, Hive, RDBMS, Kafka etc. out of the box. It also has really cool & user friendly interface which can be used to build the dataflow in minutes by dragging and dropping processors. NiFi is an alternative with more support and customer adoption. It has been used heavily at NSA and it is part of Hortonworks Data Flow. To learn more about Apache NiFi go here: https://nifi.apache.org/ NiFi tutorials here: http://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/ Falcon tutorials here: http://hortonworks.com/apache/falcon/#tutorials They do much more than a scheduler. They help build true pipelines which is the usual use case. I evaluated AirFlow and while is a promising project. It is still an incubator phase, not enterprise ready - still many issues, and it is more like a traditional scheduler. It depends on your use case, but if you want other Falcon or Nifi, you can achieve what scheduler does or more. I just love NiFi because I can use it for Hadoop and non-Hadoop. Let me know if you want to see a demo of NiFi and I can set you up.
... View more
