Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Obtain Data Ingestion lineage from Apache Nifi via Cloudera Navigator

avatar
Explorer

Hi, I am trying to obtain data ingestion lineage of Nifi via Cloudera Navigator. I have not come across any solution or documentation for the same. Has anyone experimented with this successfully? 

Is there an existing solution for this?

1 ACCEPTED SOLUTION

avatar
Master Collaborator

Yeah, thats right. Unfortunately there is no feature available to gather Nifi Lineage in Navigator. 

View solution in original post

5 REPLIES 5

avatar
Master Collaborator

At present Navigator doesn't support gathering lineage from NiFi - however within nifi there is lineage of flowfile. You can get the steps from this link https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/getting-started-with-apache-nifi/content/lin...

avatar
Explorer

Thank you for the clarification.

But I was asking more from the perspective of having 1 central place where the data lineage can be highlighted. Ingestion -> Transformation (all steps in between) -> Visualization.

avatar
Master Collaborator

Right, In CDH its not available but with CDP you have options to install Atlas which already have integration with NiFi

https://docs.cloudera.com/cdpdc/7.0/overview/topics/cdpdc-overview.html

  • Data Engineering

    Ingest, transform, and analyze data.

    Services: HDFS, YARN, YARN Queue Manager, Ranger, Atlas, Hive Metastore, Hive on Tez, Spark, Oozie, Hue, and Data Analytics Studio

  • Data Mart

    Browse, query, and explore your data in an interactive way.

    Services: HDFS, YARN, YARN Queue Manager, Ranger, Atlas, Hive Metastore, Impala, and Hue

  • Operational Database

    Low latency writes, reads, and persistent access to data for Online Transactional Processing (OLTP) use cases.

    Services: HDFS, Ranger, Atlas, and HBase

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/installing-hdf/content/configure_nifi_for_at...

avatar
Explorer

Thank you again. Yes I know we can use Atlas in CDP.

So, all in all, there is not even a workaround to get Nifi lineage on Navigator right?

avatar
Master Collaborator

Yeah, thats right. Unfortunately there is no feature available to gather Nifi Lineage in Navigator.