Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Airflow and Atlas integration

Airflow and Atlas integration

Expert Contributor

Did anyone try to configure Airflow for sending lineage metadata to Atlas?

I try to configure by this instruction: https://airflow.apache.org/lineage.html

But in Atlas web ui I see nothing about Airflow when try to search by type.

I have kerberized cluster HDP 2.6.5 and Airflow 1.10.2

99508-capture.jpg

99507-capture.jpg

3 REPLIES 3
Highlighted

Re: Airflow and Atlas integration

Mentor

@Ruslan Fialkovsky

How did you install the Airflow, I would think you need the Ambari integration for it to work.

Have a look at this git Apache Airflow management pack for Ambari after the integration and configuration that's when I think you can see the lineage in Atlas.

If you use Ambari it will generate automatically the Kerberos principal and keytab !!

Highlighted

Re: Airflow and Atlas integration

Expert Contributor
@Geoffrey Shelton Okot

I tried to install airflow by airflow-mpack but it contain many bugs. I installed airflow and generated keytab by myself it works fine.

I found some information on airflow issue tracker about this problem:

https://issues.apache.org/jira/browse/AIRFLOW-3486

In fact lineage transmission isn't working now.

Highlighted

Re: Airflow and Atlas integration

Mentor

@Ruslan Fialkovsky

There is a patch attached did you update your code?

Don't have an account?
Coming from Hortonworks? Activate your account here