Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Airflow and Atlas integration

Highlighted

Airflow and Atlas integration

Expert Contributor

Did anyone try to configure Airflow for sending lineage metadata to Atlas?

I try to configure by this instruction: https://airflow.apache.org/lineage.html

But in Atlas web ui I see nothing about Airflow when try to search by type.

I have kerberized cluster HDP 2.6.5 and Airflow 1.10.2

99508-capture.jpg

99507-capture.jpg

3 REPLIES 3

Re: Airflow and Atlas integration

Mentor

@Ruslan Fialkovsky

How did you install the Airflow, I would think you need the Ambari integration for it to work.

Have a look at this git Apache Airflow management pack for Ambari after the integration and configuration that's when I think you can see the lineage in Atlas.

If you use Ambari it will generate automatically the Kerberos principal and keytab !!

Re: Airflow and Atlas integration

Expert Contributor
@Geoffrey Shelton Okot

I tried to install airflow by airflow-mpack but it contain many bugs. I installed airflow and generated keytab by myself it works fine.

I found some information on airflow issue tracker about this problem:

https://issues.apache.org/jira/browse/AIRFLOW-3486

In fact lineage transmission isn't working now.

Re: Airflow and Atlas integration

Mentor

@Ruslan Fialkovsky

There is a patch attached did you update your code?