About dansteu

dansteu · ‎10-27-2021

Hi @slambe , still related to this thread, I would like to know how it's possible to see data lineage of an existing Hive DB. Better to explain me; if I execute the data lineage executing the scripts of the article https://community.cloudera.com/t5/Community-Articles/Using-Apache-Atlas-to-view-Data-Lineage/ta-p/246305 it works perfectly now. But imagine to suggest this solution to an existing prod environment with tons of tables, relations - the scripts of creation are hidden to me but I would guess Atlas can read table fields and dependencies; how could the data lineage be available in Atlas in this case? the logic between tables is hidden to Atlas cause the creation of structures has been defined during Hive DB creation, years ago, and it's not a live process. Thanks, Best Regards, Daniele.

dansteu · ‎10-05-2021

Hi Team, following the resolution thread Re: How to report NiFi lineage to Atlas I am now able to use the Atlas lib of Nifi called "ReportLineageToAtlas". I configured the settings as the following (Atlas URLs is changing day by day, after the restart of the servers) : In Kerberos Credential Service I inserted Atlas Kerberos Keytab and Atlas Kerberos Principal. The error I receive while the "ReportLineageToAtlas" is running is about Authentication to Atlas to reach the API : Indeed if I open the Atlas web URL : http://34.134.205.221:31000/api/atlas/v2/types/typedefs?name=nifi_output_port I have no data. Therefore we have two issues: 1) it seems there's an authentication issue from Atlas towards Nifi API. 2) it seems in the API there are no data named "nifi_output_port" (maybe because it's not yet able to authenticate?). Maybe the two issues are linked. Do you have any hints about how to fix this problem? Were am I wrong in the settings? I can provide you further details if needed. Thanks a lot! Daniele.

dansteu · ‎10-05-2021

Thanks @MattWho ! It works. The feature "ReportLineageToAtlas" is now enabled in NiFi. Thanks a lot.

dansteu · ‎10-04-2021

Hi All, I have Nifi 1.14.0 currently installed and I would like to connect this to Apache Atlas available in Cloudera Cluster. I read here Configure NiFi for Atlas Integration (cloudera.com) there should be the possibility to do this by configuring ReportLineageToAtlas task in NiFi. Unfortunately this option is missing in my NiFi installation. Do you have any hints about how to add this feature in my NiFi Centos installation? Thanks a lot for your support. Best Regards, Daniele.

dansteu · ‎09-24-2021

Hi @slambe I solved the issue and I am now able to see Data Lineage correctly. Thanks a lot for the support. Best Regards, Daniele.

dansteu · ‎09-20-2021

Hi @slambe Thanks for linking me the documentation. I followed that one in detail and now tables are automatically triggered from Hive into Atlas without executing any manual script. Thanks! Unfortunately we still face "No lineage data found". For sample I tried to execute this simple use case https://community.cloudera.com/t5/Community-Articles/Using-Apache-Atlas-to-view-Data-Lineage/ta-p/246305 but branch_intersect table is still not showing any Data Lineage. Is there any Kafka topic to be configured for Data Lineage? In the guide followed those are not mentioned but I can retrieve something about that in other Cloudera forum whenever I search for "No lineage data found". Thanks, Daniele.

dansteu · ‎09-17-2021

Hi @slambe , Thanks a lot for your assistance. So as far as I understood, only the automatic data feeding towards Atlas will integrate Data Lineage information. Could you please link me any official Cloudera documentation in order to set up this Hive Hook for Atlas? I mean hive-site.xml settings and any other configuration we need to care also in Kafka for sample. Thanks a lot for your support. Best Regards, Daniele.

dansteu · ‎09-15-2021

Hi Support team, I kindly need your support. I am doing some tests in order to evaluate into Cloudera the Atlas data lineage by importing Hive DB. I executed several tests and also followed entirely this sample: https://community.cloudera.com/t5/Community-Articles/Using-Apache-Atlas-to-view-Data-Lineage/ta-p/246305 In order to migrate all the entities into Atlas, I successfully executed the import-hive.sh script that is located in Atlas hook-bin folder. The result of the migration shows (into Atlas) Hive test DB with all its entities but I cannot see in Atlas any lineage (that should appear as the join between two tables in order to create the third one), simply the message: " No lineage data found ". I also checked that if I create EXTERNAL TABLES rather than TABLE, something appears in Lineage section but is incomplete and not showing exactly the dependencies with source table names. Could you please guide me in the solution of this issue that is hiding the Data Lineage? I checked also logs here located /var/logs/atlas/application.log and all seem to be fine except a strange error " ERROR - [etp1881561036-238:] ~ Exception while fetching groups (AtlasAbstractAuthenticationProvider:137)java.io.IOException: No groups found for user HTTP " that seem to be not linked to this issue. I followed the hints of this thread "https://cloudera.ericlin.me/2018/08/webhcat-request-failed-with-error-id-http-no-such-user/" to fix this other error message but it's still appearing. Thanks a lot in advance for your support. Best Regards

Online	Offline
Last Visited	‎11-16-2021 04:30 AM

Member Since	‎09-14-2021 09:04 AM
Last Visited	‎11-16-2021 04:30 AM
Posts	19

Cloudera Community

Re: How to report NiFi lineage to Atlas

Re: Data Lineage not shown after importing Hive en...

Authentication issue while connecting Nifi to Atla...

Re: How to report NiFi lineage to Atlas

How to report NiFi lineage to Atlas

Re: Data Lineage not shown after importing Hive en...

Re: Data Lineage not shown after importing Hive en...

Re: Data Lineage not shown after importing Hive en...

Data Lineage not shown after importing Hive entiti...