About dansteu

dansteu · ‎10-27-2021

Hi @slambe , still related to this thread, I would like to know how it's possible to see data lineage of an existing Hive DB. Better to explain me; if I execute the data lineage executing the scripts of the article https://community.cloudera.com/t5/Community-Articles/Using-Apache-Atlas-to-view-Data-Lineage/ta-p/246305 it works perfectly now. But imagine to suggest this solution to an existing prod environment with tons of tables, relations - the scripts of creation are hidden to me but I would guess Atlas can read table fields and dependencies; how could the data lineage be available in Atlas in this case? the logic between tables is hidden to Atlas cause the creation of structures has been defined during Hive DB creation, years ago, and it's not a live process. Thanks, Best Regards, Daniele.

dansteu · ‎10-11-2021

Hi Support Team, I am facing the issue of timeout while sending data to Kafka from Nifi. In Nifi I am using Publish_Kafka_1_0 transformer configured like this : I tested previously the connection between Nifi server and Kafka server towards 9092 port and it's fine. Unfortunatly while executing the transformer, the following error is appearing: Failed to send StandardFlowFileRecord[uuid=b01bfeb3-3ce7-42c2-9abb-9a9f9930b87c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1633677179202-1, container=default, section=1], offset=20, length=4], offset=0,name=b01bfeb3-3ce7-42c2-9abb-9a9f9930b87c,size=4] to Kafka: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 20000 ms. The kerberos connection is totally fine, so I don't know which is the connection bottleneck. Can you kindly provide any hints ? Thanks, Daniele.

dansteu · ‎10-05-2021

Hi Team, following the resolution thread Re: How to report NiFi lineage to Atlas I am now able to use the Atlas lib of Nifi called "ReportLineageToAtlas". I configured the settings as the following (Atlas URLs is changing day by day, after the restart of the servers) : In Kerberos Credential Service I inserted Atlas Kerberos Keytab and Atlas Kerberos Principal. The error I receive while the "ReportLineageToAtlas" is running is about Authentication to Atlas to reach the API : Indeed if I open the Atlas web URL : http://34.134.205.221:31000/api/atlas/v2/types/typedefs?name=nifi_output_port I have no data. Therefore we have two issues: 1) it seems there's an authentication issue from Atlas towards Nifi API. 2) it seems in the API there are no data named "nifi_output_port" (maybe because it's not yet able to authenticate?). Maybe the two issues are linked. Do you have any hints about how to fix this problem? Were am I wrong in the settings? I can provide you further details if needed. Thanks a lot! Daniele.

dansteu · ‎10-05-2021

Thanks @MattWho ! It works. The feature "ReportLineageToAtlas" is now enabled in NiFi. Thanks a lot.

dansteu · ‎10-04-2021

Hello Team, @ahadjidj Running this suggested command mvn clean install -Pinclude-atlas -DskipTests referrinng to the pom xml file here located: /work/nar/framework/nifi-framework-nar-1.14.0.nar-unpacked/META-INF/maven/org.apache.nifi/nifi-framework-nar/pom.xml I get this error message : [WARNING] The requested profile "include-atlas" could not be activated because it does not exist. [ERROR] Failed to execute goal org.apache.nifi:nifi-nar-maven-plugin:1.3.1:nar (default-nar) on project nifi-evtx-nar: The plugin org.apache.nifi:nifi-nar-maven-plugin:1.3.1 requires Maven vers ion 3.1.0 -> [Help 1] I got this message: The requested profile "include-atlas" could not be activated because it does not exist. Could you please provide any hint? As we need to introduce the lineage into Atlas with info coming from Nifi, this area in currently on priority as we are stucked on this. Thanks a lot! Daniele.

dansteu · ‎10-04-2021

Hi All, I have Nifi 1.14.0 currently installed and I would like to connect this to Apache Atlas available in Cloudera Cluster. I read here Configure NiFi for Atlas Integration (cloudera.com) there should be the possibility to do this by configuring ReportLineageToAtlas task in NiFi. Unfortunately this option is missing in my NiFi installation. Do you have any hints about how to add this feature in my NiFi Centos installation? Thanks a lot for your support. Best Regards, Daniele.

dansteu · ‎09-24-2021

Hi @slambe I solved the issue and I am now able to see Data Lineage correctly. Thanks a lot for the support. Best Regards, Daniele.

dansteu · ‎09-20-2021

Hi @slambe Thanks for linking me the documentation. I followed that one in detail and now tables are automatically triggered from Hive into Atlas without executing any manual script. Thanks! Unfortunately we still face "No lineage data found". For sample I tried to execute this simple use case https://community.cloudera.com/t5/Community-Articles/Using-Apache-Atlas-to-view-Data-Lineage/ta-p/246305 but branch_intersect table is still not showing any Data Lineage. Is there any Kafka topic to be configured for Data Lineage? In the guide followed those are not mentioned but I can retrieve something about that in other Cloudera forum whenever I search for "No lineage data found". Thanks, Daniele.

dansteu · ‎09-17-2021

Hi @slambe , Thanks a lot for your assistance. So as far as I understood, only the automatic data feeding towards Atlas will integrate Data Lineage information. Could you please link me any official Cloudera documentation in order to set up this Hive Hook for Atlas? I mean hive-site.xml settings and any other configuration we need to care also in Kafka for sample. Thanks a lot for your support. Best Regards, Daniele.

dansteu · ‎09-16-2021

Hi All, I want to integrate my Atlas installation available in Cloudera with external Nifi Cluster , which is outside the Hortonworks Data Platform. Would you please support me to know whether it's possible or it is mandatory that the Nifi cluster needs to exist within the HDP itself ? Thanks a lot for your support, Daniele

Online	Offline
Last Visited	‎11-16-2021 04:30 AM

Member Since	‎09-14-2021 09:04 AM
Last Visited	‎11-16-2021 04:30 AM
Posts	19

Cloudera Community

Re: How to report NiFi lineage to Atlas

Re: Data Lineage not shown after importing Hive en...

Unable to send data to Kafka from Nifi

Authentication issue while connecting Nifi to Atla...

Re: How to report NiFi lineage to Atlas

Re: Reporting NiFi lineage to Atlas

How to report NiFi lineage to Atlas

Re: Data Lineage not shown after importing Hive en...

Re: Data Lineage not shown after importing Hive en...

Re: Data Lineage not shown after importing Hive en...

Can we integrate an external Nifi cluster with Atl...