Obtain Data Ingestion lineage from Apache Nifi via Cloudera Navigator

[#1]

Hi, I am trying to obtain data ingestion lineage of Nifi via Cloudera Navigator. I have not come across any solution or documentation for the same. Has anyone experimented with this successfully? 

Is there an existing solution for this?

[#2]

At present Navigator doesn't support gathering lineage from NiFi - however within nifi there is lineage of flowfile. You can get the steps from this link https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/getting-started-with-apache-nifi/content/lin...

[#3]

Thank you for the clarification.

But I was asking more from the perspective of having 1 central place where the data lineage can be highlighted. Ingestion -> Transformation (all steps in between) -> Visualization.

[#4]

Right, In CDH its not available but with CDP you have options to install Atlas which already have integration with NiFi

https://docs.cloudera.com/cdpdc/7.0/overview/topics/cdpdc-overview.html

• Data Engineering

  Ingest, transform, and analyze data.

  Services: HDFS, YARN, YARN Queue Manager, Ranger, Atlas, Hive Metastore, Hive on Tez, Spark, Oozie, Hue, and Data Analytics Studio

• Data Mart

  Browse, query, and explore your data in an interactive way.

  Services: HDFS, YARN, YARN Queue Manager, Ranger, Atlas, Hive Metastore, Impala, and Hue

• Operational Database

  Low latency writes, reads, and persistent access to data for Online Transactional Processing (OLTP) use cases.

  Services: HDFS, Ranger, Atlas, and HBase

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/installing-hdf/content/configure_nifi_for_at...

[#5]

Thank you again. Yes I know we can use Atlas in CDP.

So, all in all, there is not even a workaround to get Nifi lineage on Navigator right?

[#6 · Accepted solution]

Yeah, thats right. Unfortunately there is no feature available to gather Nifi Lineage in Navigator.