Obtain Data Ingestion lineage from Apache Nifi via Cloudera Navigator

5 replies
4856 views
Translate All

Hi, I am trying to obtain data ingestion lineage of Nifi via Cloudera Navigator. I have not come across any solution or documentation for the same. Has anyone experimented with this successfully?

Is there an existing solution for this?

venkatsambathAnswer

Gold ⭐️⭐️⭐️

Yeah, thats right. Unfortunately there is no feature available to gather Nifi Lineage in Navigator.

venkatsambath

Gold ⭐️⭐️⭐️

At present Navigator doesn't support gathering lineage from NiFi - however within nifi there is lineage of flowfile. You can get the steps from this link https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/getting-started-with-apache-nifi/content/lineage-graph.html

Translate

shettyrishAuthor

Silver ⭐️⭐️

Thank you for the clarification.

But I was asking more from the perspective of having 1 central place where the data lineage can be highlighted. Ingestion -> Transformation (all steps in between) -> Visualization.

Translate

venkatsambath

Gold ⭐️⭐️⭐️

Right, In CDH its not available but with CDP you have options to install Atlas which already have integration with NiFi

https://docs.cloudera.com/cdpdc/7.0/overview/topics/cdpdc-overview.html

Data Engineering
Ingest, transform, and analyze data.
Services: HDFS, YARN, YARN Queue Manager, Ranger, Atlas, Hive Metastore, Hive on Tez, Spark, Oozie, Hue, and Data Analytics Studio
Data Mart
Browse, query, and explore your data in an interactive way.
Services: HDFS, YARN, YARN Queue Manager, Ranger, Atlas, Hive Metastore, Impala, and Hue
Operational Database
Low latency writes, reads, and persistent access to data for Online Transactional Processing (OLTP) use cases.
Services: HDFS, Ranger, Atlas, and HBase

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/installing-hdf/content/configure_nifi_for_atlas_integration.html

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.