Skip to main content
dansteu
dansteu
Gold ⭐️
September 15, 2021
Solved

Data Lineage not shown after importing Hive entities

Hi Support team,

I kindly need your support.

I am doing some tests in order to evaluate into Cloudera the Atlas data lineage by importing Hive DB.

I executed several tests and also followed entirely this sample: https://community.cloudera.com/t5/Community-Articles/Using-Apache-Atlas-to-view-Data-Lineage/ta-p/246305

 

In order to migrate all the entities into Atlas, I successfully executed the import-hive.sh script that is located in Atlas hook-bin folder.

The result of the migration shows (into Atlas) Hive test DB with all its entities but I cannot see in Atlas any lineage (that should appear as the join between two tables in order to create the third one), simply the message: " No lineage data found ".
I also checked that if I create EXTERNAL TABLES rather than TABLE, something appears in Lineage section but is incomplete and not showing exactly the dependencies with source table names.

Could you please guide me in the solution of this issue that is hiding the Data Lineage?

 

I checked also logs here located /var/logs/atlas/application.log and all seem to be fine except a strange error " ERROR - [etp1881561036-238:] ~ Exception while fetching groups (AtlasAbstractAuthenticationProvider:137)java.io.IOException: No groups found for user HTTP "

that seem to be not linked to this issue. I followed the hints of this thread "https://cloudera.ericlin.me/2018/08/webhcat-request-failed-with-error-id-http-no-such-user/" to fix this other error message but it's still appearing.

 

Thanks a lot in advance for your support.

Best Regards

 

 

 

 

 

slambe
slambeAnswer
Gold ⭐️
September 16, 2021

Hi @dansteu,

 

First of all, executing the import-hive.sh command will only import Hive entities (DBs, tables, columns) into Atlas. It won't create the inter-table lineage.

 

The only way to see lineage is for Atlas to automatically fetch those entities from Hive using the Atlas hook. In order to see lineage, you need the following components in a healthy state: Hive, Hbase, Kafka, Solr, Atlas.

 

The article you referred to is correct and note that it doesn't ask you to run import-hive.sh at all! Hope this helps.

 

Regards,

Saurabh

View in thread

8 replies

slambe
Gold ⭐️
slambeAnswer
Gold ⭐️
September 16, 2021

Hi @dansteu,

 

First of all, executing the import-hive.sh command will only import Hive entities (DBs, tables, columns) into Atlas. It won't create the inter-table lineage.

 

The only way to see lineage is for Atlas to automatically fetch those entities from Hive using the Atlas hook. In order to see lineage, you need the following components in a healthy state: Hive, Hbase, Kafka, Solr, Atlas.

 

The article you referred to is correct and note that it doesn't ask you to run import-hive.sh at all! Hope this helps.

 

Regards,

Saurabh

Translate
dansteu
dansteuAuthor
Gold ⭐️
September 17, 2021

Hi @slambe ,

Thanks a lot for your assistance.

So as far as I understood, only the automatic data feeding towards Atlas will integrate Data Lineage information.

Could you please link me any official Cloudera documentation in order to set up this Hive Hook for Atlas? I mean hive-site.xml settings and any other configuration we need to care also in Kafka for sample.

 

Thanks a lot for your support.

Best Regards,

Daniele.

Translate
slambe
Gold ⭐️
slambe
Gold ⭐️
September 17, 2021

@dansteuyou are correct. Only if Atlas fetches Hive metadata automatically (with no manual intervention), only then the lineage is created.

Please refer to this Apache Atlas document to set up the Atlas Hive hook:
https://atlas.apache.org/1.2.0/Hook-Hive.html

BTW, when you install Atlas in an HDP cluster, these properties are automatically set and no intervention is needed. So you might just need to confirm if those properties are set.

 

Regards,

Saurabh

Translate
Terms & ConditionsAccessibility statement