Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Lineage is not visible for Hive Table in Atlas

Highlighted

Re: Lineage is not visible for Hive Table in Atlas

New Contributor

95413-hook.png

@Sharmadha Sainath

Hive Hook is enabled Please find attachment for your reference.

Checked hive permission and Atlas that is perfect no issues in Permission.

Still when we are creating new tables it is not reflecting in Atlas ui. we need to run import-hive.sh.

We are getting below error when we run import-hive.sh script. it is coming in application.log in Atlas rest services has no errors in their logs.

This errors are for newly created table. but still due to this error also we are able to see external table and its lineage.

2018-12-13 04:32:58,136 ERROR - [pool-2-thread-9 - 04e4ba41-c09d-4257-9391-1fde800fa5d2:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.dep13@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:32:59,132 ERROR - [pool-2-thread-10 - fd8d6366-e465-450a-a42a-8d98022787f2:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.dep2@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:01,145 ERROR - [pool-2-thread-9 - bceb4f09-1cce-461d-8b1a-2e4a90e39717:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.dep2@C03:1544606069000} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:04,374 ERROR - [pool-2-thread-7 - e62c07bc-4fe4-480a-bf80-6ffff789c698:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee19@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:04,975 ERROR - [pool-2-thread-8 - d2f31e6c-fe99-40d8-a153-087b9c10eb8b:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee20@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:05,347 ERROR - [pool-2-thread-9 - 4f478d9e-4c32-456e-8669-0d4bcef9bcf8:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data0@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:06,139 ERROR - [pool-2-thread-9 - e6da522f-ff85-436b-b66b-2844ca2fdf2c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data04@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:06,388 ERROR - [pool-2-thread-8 - a385f421-d818-4fc6-a613-880064babcf9:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.employee_data04@C03:1544688356000} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:07,468 ERROR - [pool-2-thread-10 - e5c8525c-7877-4e1a-b2e6-b23d9f054e94:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data18@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:07,959 ERROR - [pool-2-thread-9 - 72143f28-2630-4306-8639-7c513dffc51c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.employee_data2@C03:1544605910000} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:08,180 ERROR - [pool-2-thread-10 - 516cb986-69ae-48f9-813f-3e7b23a1b967:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data20@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:08,439 ERROR - [pool-2-thread-9 - 5706eb89-2951-4c33-b914-9ace101cb011:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data3@C03} does not exist (GraphTransactionInterceptor:73)

After running import script we are able to see only External tables in Atlas. no lineage for Managed and View tables.

Could you please help me on this.

Thanks,

Owez

Re: Lineage is not visible for Hive Table in Atlas

Expert Contributor

@Owez Mujawar

What is the offsets.topic.replication.factor in kafka properties ? Is it greater than the number of kafka brokers ? Can you check kafka logs for error messages like the following ?

ERROR [KafkaApi-1001] Number of alive brokers 'x' does not meet the required replication factor 'x' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

Re: Lineage is not visible for Hive Table in Atlas

New Contributor

@Sharmadha Sainath

Please find attachment for your reference. We have offsets.topic.replication.factor=1 and we have 1 Kafka broker.

We haven't found any error in Kafka logs.

97381-offset.png

97382-kafka.png

KAFKA Logs:

[2018-12-13 04:29:43,286] INFO Updated PartitionLeaderEpoch. New: {epoch:16, offset:1063289}, Current: {epoch:14, offset916036} for Partition: ATLAS_ENTITIES-0. Cache now contains 8 entries. (kafka.server.epoch.LeaderEpochFileCache)
[2018-12-13 04:34:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2018-12-13 04:44:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2018-12-13 04:54:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)

Still we haven't fount any solution. Could you please help me to resolved this issue on priority.

Thanks,

Owez

Re: Lineage is not visible for Hive Table in Atlas

Expert Contributor
@Owez Mujawar

Could you run kafka console consumer on topic ATLAS_HOOK and ATLAS_ENTITIES when you create a table and check if the messages are flowing to the topic ?