Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

Airflow and Atlas integration

Expert Contributor

Did anyone try to configure Airflow for sending lineage metadata to Atlas?

I try to configure by this instruction: https://airflow.apache.org/lineage.html

But in Atlas web ui I see nothing about Airflow when try to search by type.

I have kerberized cluster HDP 2.6.5 and Airflow 1.10.2

99508-capture.jpg

99507-capture.jpg

3 REPLIES 3

Mentor

@Ruslan Fialkovsky

How did you install the Airflow, I would think you need the Ambari integration for it to work.

Have a look at this git Apache Airflow management pack for Ambari after the integration and configuration that's when I think you can see the lineage in Atlas.

If you use Ambari it will generate automatically the Kerberos principal and keytab !!

Expert Contributor
@Geoffrey Shelton Okot

I tried to install airflow by airflow-mpack but it contain many bugs. I installed airflow and generated keytab by myself it works fine.

I found some information on airflow issue tracker about this problem:

https://issues.apache.org/jira/browse/AIRFLOW-3486

In fact lineage transmission isn't working now.

Mentor

@Ruslan Fialkovsky

There is a patch attached did you update your code?