About dansteu

dansteu · ‎10-27-2021

Hi @slambe , still related to this thread, I would like to know how it's possible to see data lineage of an existing Hive DB. Better to explain me; if I execute the data lineage executing the scripts of the article https://community.cloudera.com/t5/Community-Articles/Using-Apache-Atlas-to-view-Data-Lineage/ta-p/246305 it works perfectly now. But imagine to suggest this solution to an existing prod environment with tons of tables, relations - the scripts of creation are hidden to me but I would guess Atlas can read table fields and dependencies; how could the data lineage be available in Atlas in this case? the logic between tables is hidden to Atlas cause the creation of structures has been defined during Hive DB creation, years ago, and it's not a live process. Thanks, Best Regards, Daniele.

Wynner · ‎10-05-2021

@dansteu There are multiple changes needed in the properties. The Atlas Configuration Directory shouldn't be on the /tmp directory. The ideal location would be a directory in the default class path of NiFi. Create Atlas Configuration File should be set to true If Kerberos is being used for authentication, why is the URL http and not https? For the NIFi URL for Atlas, it would be better to use http://${hostname(true)}:8080/nifi this will provide a URL that will map back to the NiFi component that generated the Atlas lineage The Atlas Default Metadata Namespace should be populated with a value that is representative of the flow. For example, For example, hdf352 or testflowversion1, or something similar.

dansteu · ‎10-05-2021

Thanks @MattWho ! It works. The feature "ReportLineageToAtlas" is now enabled in NiFi. Thanks a lot.

Online	Offline
Last Visited	‎11-16-2021 04:30 AM

Member Since	‎09-14-2021 09:04 AM
Last Visited	‎11-16-2021 04:30 AM
Posts	19

Cloudera Community

Re: How to report NiFi lineage to Atlas

Re: Data Lineage not shown after importing Hive en...

Re: Authentication issue while connecting Nifi to ...

Re: How to report NiFi lineage to Atlas