Re: How to report NiFi lineage to Atlas

ManuelCalvo · ‎11-29-2021

Hi @dansteu The kerberos service name property has to be the service name specified for the kafka service which is usually "kafka".

dansteu · ‎11-02-2021

Hi All, I am following a project where we need to follow some constraint about Kafka. In particular we are expected to use the existing Kafka service installed in a separate cluster, rather than installing a new Kafka service in Cloudera cluster being used by Atlas. Is it possible to configure Atlas to use external Kafka installation, do you kindly have any suggestions on regard? Can you confirm Kafka is fundamental for Atlas and there's no way to avoid its usage? Thanks a lot, Best Regards, Daniele.

dansteu · ‎10-27-2021

Hi @slambe , still related to this thread, I would like to know how it's possible to see data lineage of an existing Hive DB. Better to explain me; if I execute the data lineage executing the scripts of the article https://community.cloudera.com/t5/Community-Articles/Using-Apache-Atlas-to-view-Data-Lineage/ta-p/246305 it works perfectly now. But imagine to suggest this solution to an existing prod environment with tons of tables, relations - the scripts of creation are hidden to me but I would guess Atlas can read table fields and dependencies; how could the data lineage be available in Atlas in this case? the logic between tables is hidden to Atlas cause the creation of structures has been defined during Hive DB creation, years ago, and it's not a live process. Thanks, Best Regards, Daniele.

Wynner · ‎10-05-2021

@dansteu There are multiple changes needed in the properties. The Atlas Configuration Directory shouldn't be on the /tmp directory. The ideal location would be a directory in the default class path of NiFi. Create Atlas Configuration File should be set to true If Kerberos is being used for authentication, why is the URL http and not https? For the NIFi URL for Atlas, it would be better to use http://${hostname(true)}:8080/nifi this will provide a URL that will map back to the NiFi component that generated the Atlas lineage The Atlas Default Metadata Namespace should be populated with a value that is representative of the flow. For example, For example, hdf352 or testflowversion1, or something similar.

dansteu · ‎10-05-2021

Thanks @MattWho ! It works. The feature "ReportLineageToAtlas" is now enabled in NiFi. Thanks a lot.

dansteu · ‎10-04-2021

Hello Team, @ahadjidj Running this suggested command mvn clean install -Pinclude-atlas -DskipTests referrinng to the pom xml file here located: /work/nar/framework/nifi-framework-nar-1.14.0.nar-unpacked/META-INF/maven/org.apache.nifi/nifi-framework-nar/pom.xml I get this error message : [WARNING] The requested profile "include-atlas" could not be activated because it does not exist. [ERROR] Failed to execute goal org.apache.nifi:nifi-nar-maven-plugin:1.3.1:nar (default-nar) on project nifi-evtx-nar: The plugin org.apache.nifi:nifi-nar-maven-plugin:1.3.1 requires Maven vers ion 3.1.0 -> [Help 1] I got this message: The requested profile "include-atlas" could not be activated because it does not exist. Could you please provide any hint? As we need to introduce the lineage into Atlas with info coming from Nifi, this area in currently on priority as we are stucked on this. Thanks a lot! Daniele.

ShounenG · ‎09-16-2021

The answer would be no. @dansteu I once came into a support case. A customer wanted to use NiFi to integrate with an external Ranger, which means that NiFi and Ranger are not under the same Ambari management. Ranger's SME told us that this is impossible. At least this is not within the scope of Cloudera's technical support. The logic should be the same for Atlas. However, if you know enough about Ranger and Ambari, it should still be possible technically in theory.

dansteu · ‎09-16-2021

Hi, I created yesterday also this new thread https://community.cloudera.com/t5/Support-Questions/Data-Lineage-not-shown-after-importing-Hive-entities/td-p/324441 Thanks in advance for support. Best Regards, Daniele.

dansteu · ‎09-15-2021

Hi, I am getting exactly same issue. As a consequence, the data lineage is not shown in Atlas after import-hive.sh script is executed. Did you resolve this topic? Could you please give me any hint? Thanks a lot!

dansteu · ‎09-14-2021

Hi, I know this question is an old one, but I am exactly in this situation. I have Cloudera and Hive that has been filled through Hue. I created a simple data structure with a couple of tables, then joined them in order to create a third table. I then run the script to import these data into Atlas. I figure out now in this forum that import-hive script is not supporting data lineage. Indeed after script execution, data lineage for tables is empty. How can I then migrate data from Hive into Atlas, including the data lineage? Thanks a lot for your answer.

Online	Offline
Last Visited	‎11-16-2021 04:30 AM

Date Registered	‎09-14-2021 09:04 AM
Last Visited	‎11-16-2021 04:30 AM
Total Messages Posted	19

Re: How to report NiFi lineage to Atlas

Re: Unable to send data to Kafka from Nifi

Configure Atlas in Cloudera Cluster using external...

Re: Data Lineage not shown after importing Hive en...

Re: Authentication issue while connecting Nifi to ...

Re: How to report NiFi lineage to Atlas

Re: Reporting NiFi lineage to Atlas

Re: Can we integrate an external Nifi cluster with...

Re: Apache Atlas Hive hook and data lineage troubl...

Re: Atlas Lineage Issue

Re: I run the script/tool(import-hive.sh) and i ca...