question Re: Obatin Data Ingestion lineage from Apache Nifi via Cloudera Navigator in Support Questions

Obtain Data Ingestion lineage from Apache Nifi via Cloudera Navigator

shettyrish — Mon, 16 Mar 2020 09:24:07 GMT

Hi, I am trying to obtain data ingestion lineage of Nifi via Cloudera Navigator. I have not come across any solution or documentation for the same. Has anyone experimented with this successfully?

Is there an existing solution for this?

Re: Obatin Data Ingestion lineage from Apache Nifi via Cloudera Navigator

venkatsambath — Mon, 16 Mar 2020 08:09:52 GMT

At present Navigator doesn't support gathering lineage from NiFi - however within nifi there is lineage of flowfile. You can get the steps from this link https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/getting-started-with-apache-nifi/content/lineage-graph.html

Re: Obatin Data Ingestion lineage from Apache Nifi via Cloudera Navigator

shettyrish — Mon, 16 Mar 2020 08:14:07 GMT

Thank you for the clarification.

But I was asking more from the perspective of having 1 central place where the data lineage can be highlighted. Ingestion -> Transformation (all steps in between) -> Visualization.

Re: Obatin Data Ingestion lineage from Apache Nifi via Cloudera Navigator

venkatsambath — Mon, 16 Mar 2020 08:22:49 GMT

Right, In CDH its not available but with CDP you have options to install Atlas which already have integration with NiFi

https://docs.cloudera.com/cdpdc/7.0/overview/topics/cdpdc-overview.html

Data Engineering
Ingest, transform, and analyze data.
Services: HDFS, YARN, YARN Queue Manager, Ranger, Atlas, Hive Metastore, Hive on Tez, Spark, Oozie, Hue, and Data Analytics Studio
Data Mart
Browse, query, and explore your data in an interactive way.
Services: HDFS, YARN, YARN Queue Manager, Ranger, Atlas, Hive Metastore, Impala, and Hue
Operational Database
Low latency writes, reads, and persistent access to data for Online Transactional Processing (OLTP) use cases.
Services: HDFS, Ranger, Atlas, and HBase

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/installing-hdf/content/configure_nifi_for_atlas_integration.html

Re: Obatin Data Ingestion lineage from Apache Nifi via Cloudera Navigator

shettyrish — Mon, 16 Mar 2020 08:30:33 GMT

Thank you again. Yes I know we can use Atlas in CDP.

So, all in all, there is not even a workaround to get Nifi lineage on Navigator right?

Re: Obatin Data Ingestion lineage from Apache Nifi via Cloudera Navigator

venkatsambath — Mon, 16 Mar 2020 08:44:23 GMT

Yeah, thats right. Unfortunately there is no feature available to gather Nifi Lineage in Navigator.