CDE Airflow for CML Pipeline Orchestration

Apache Airflow and Spark running on K8s provide unreasonable scalability.  Cloudera Machine Learning also runs on K8s and often the ML users leverage Apache Airflow to orchestrate their pipelines. Apache Airflow within Cloudera Data Engineering now has the capability to orchestrate CML jobs via the new Airflow CML Operator.  This time instead of writing up a detailed article, I recorded a live demo.

Assets used in the demo

• CML Operator
  • https://github.com/curtishoward/cml_operator/
• Airflow Dag
  • https://github.com/sunileman/cde-airflow-branch-operator
• Demo
  • https://www.youtube.com/watch?v=n9o_Qimnggo

Enjoy!