Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Is cloudera supports apache airflow?

avatar
New Contributor

We are planing to migrate oozie to airflow? will cloudera supports airflow 

5 REPLIES 5

avatar
Master Guru

@SVK NO. Airflow is not a part of the Cloudera suite of application. While you can definitely use 3rd party CSDs in general [1], or make your own CSDs, the support of it is outside Cloudera's scope.

If you do pursue to install, my suggestion is to install in a dev/test environment and test it out. Once you are comfortable with the product then you can install into your production cluster.

 

Again this is not under Cloudera Support's Scope. 

Feel free to reach out to your accounts team if you need more information about Cloudera's offerings.

 

[1] https://github.com/teamclairvoyant/apache-airflow-cloudera-csd


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Contributor

Although not available in CDP Base.  Airflow is provided through our  Data Engineering Experience (CDE) within Public Cloud and Private Cloud.

https://docs.cloudera.com/data-engineering/cloud/manage-jobs/topics/cde-airflow-dag-pipeline.html

https://www.cloudera.com/products/data-engineering.html?tab=0 

 

avatar
Expert Contributor

The latest document about Airflow in Data Engineering Experience (CDE) within Public Cloud and CDP Private Cloud Data Service is here:

 

 

avatar
Super Collaborator

Hello @SVK 

 

Thanks for using Cloudera Community. Based on your post, You wish to confirm if Airflow is Supported by Cloudera. Summarizing the responses shared by my Colleagues for you:

 

(I) Airflow isn't Supported by Cloudera in Standalone Mode,

(II) In CDP Public Cloud [1] & CDP Private Cloud [2], CDE (Cloudera Data Engineering) uses Airflow. Any issues encountered with CDE Airflow on CDP Public Cloud & CDP Private Cloud is Supported. 

(III) CDE allows External Airflow to be used as well. Yet, Supportability is restricted to Cloudera Airflow Providers only. Refer [3].

 

If your queries are addressed, Feel free to mark the Post as Solved. 

 

Regards, Smarak

 

[1] https://docs.cloudera.com/data-engineering/cloud/orchestrate-workflows/topics/cde-airflow-dag-pipeli...

[2] https://docs.cloudera.com/data-engineering/1.3.4/orchestrate-workflows/topics/cde-airflow-dag-pipeli...

[3] https://docs.cloudera.com/data-engineering/cloud/orchestrate-workflows/topics/cde-airflow-provider.h...

 

 

 

avatar
Super Collaborator

Hello @SVK 

 

If your queries concerning Apache Airflow has been addressed, Feel free to mark the Post as Solved. If you have any further ask, Kindly share the same & we shall get back to you accordingly. 

 

Regards, Smarak