@Vinay Could you please check your ambari-agent and ambari-server version. check in database as well. This issue occurs when there are incorrect entries present in the ambari database. The following is the table in the database. From the host_version ambari=>select*from host_version; Thanks, Owez ... View more

@Sharmadha Sainath Please find attachment for your reference. We have offsets.topic.replication.factor=1 and we have 1 Kafka broker. We haven't found any error in Kafka logs. KAFKA Logs: [2018-12-13 04:29:43,286] INFO Updated PartitionLeaderEpoch. New: {epoch:16, offset:1063289}, Current: {epoch:14, offset916036} for Partition: ATLAS_ENTITIES-0. Cache now contains 8 entries. (kafka.server.epoch.LeaderEpochFileCache) [2018-12-13 04:34:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) [2018-12-13 04:44:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) [2018-12-13 04:54:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) Still we haven't fount any solution. Could you please help me to resolved this issue on priority. Thanks, Owez ... View more

@Sharmadha Sainath Hive Hook is enabled Please find attachment for your reference. Checked hive permission and Atlas that is perfect no issues in Permission. Still when we are creating new tables it is not reflecting in Atlas ui. we need to run import-hive.sh. We are getting below error when we run import-hive.sh script. it is coming in application.log in Atlas rest services has no errors in their logs. This errors are for newly created table. but still due to this error also we are able to see external table and its lineage. 2018-12-13 04:32:58,136 ERROR - [pool-2-thread-9 - 04e4ba41-c09d-4257-9391-1fde800fa5d2:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.dep13@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:32:59,132 ERROR - [pool-2-thread-10 - fd8d6366-e465-450a-a42a-8d98022787f2:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.dep2@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:01,145 ERROR - [pool-2-thread-9 - bceb4f09-1cce-461d-8b1a-2e4a90e39717:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.dep2@C03:1544606069000} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:04,374 ERROR - [pool-2-thread-7 - e62c07bc-4fe4-480a-bf80-6ffff789c698:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee19@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:04,975 ERROR - [pool-2-thread-8 - d2f31e6c-fe99-40d8-a153-087b9c10eb8b:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee20@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:05,347 ERROR - [pool-2-thread-9 - 4f478d9e-4c32-456e-8669-0d4bcef9bcf8:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data0@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:06,139 ERROR - [pool-2-thread-9 - e6da522f-ff85-436b-b66b-2844ca2fdf2c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data04@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:06,388 ERROR - [pool-2-thread-8 - a385f421-d818-4fc6-a613-880064babcf9:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.employee_data04@C03:1544688356000} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:07,468 ERROR - [pool-2-thread-10 - e5c8525c-7877-4e1a-b2e6-b23d9f054e94:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data18@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:07,959 ERROR - [pool-2-thread-9 - 72143f28-2630-4306-8639-7c513dffc51c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.employee_data2@C03:1544605910000} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:08,180 ERROR - [pool-2-thread-10 - 516cb986-69ae-48f9-813f-3e7b23a1b967:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data20@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:08,439 ERROR - [pool-2-thread-9 - 5706eb89-2951-4c33-b914-9ace101cb011:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data3@C03} does not exist (GraphTransactionInterceptor:73) After running import script we are able to see only External tables in Atlas. no lineage for Managed and View tables. Could you please help me on this. Thanks, Owez ... View more

@Sharmadha Sainath Thanks for the confirmation. The scenario you explain that's perfect suits in our environment, but 3rd Scenario is not working in our environment. for example we are using below query to create Managed,View and External table: CREATE TABLE IF NOT EXISTS employee13 ( eid int, name int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/hive/data/employee13'; CREATE VIEW employee_data AS SELECT * FROM employee WHERE eid >0; CREATE EXTERNAL TABLE employee_data2 ( eid int, name int ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' LOCATION '/hive/data/employee'; When we execute above query its gets successfully executed. we don't see any errors in logs. When we login to Atlas UI we are not able to see that newly created tables in 1st Attempt. After running import-hive.sh script we are able to see the lineage of External tables only. We are not able see lineage of newly created managed and view tables.it comes as no linage found. Could you please guide me on that. ... View more

@Sharmadha Sainath Hive Hook is enabled and we can see all tables and linage of External table only. We want to see lineage of Managed and Virtual view tables which were created before atlas. Could you please help me to see that lineage of Managed and Virtual view tables. i have not seen any error in ranger-log , kafka-log , and hive-log. only in application-log i can see below error. 2018-12-11 00:12:48,690 ERROR - [pool-2-thread-7 - 3b35712a-6c84-4102-a19c-cedf26ea5e1c:] ~ graph rollback due to exception AtlasBaseException:Instance __AtlasUserProfile with unique attribute {name=admin} does not exist (GraphTransactionInterceptor:73) 2018-12-11 00:11:19,247 INFO - [main-SendThread(hadmgrndcc03-1.lifeway.org:2181):] ~ Opening socket connection to server hadmgrndcc03-1.lifeway.org/172.17.20.29:2181. Will not attempt to authenticate using SASL (unknown error) (ClientCnxn:1019) 2018-12-11 00:11:30,951 INFO - [main-SendThread(hadmgrndcc03-3.lifeway.org:2181):] ~ Opening socket connection to server hadmgrndcc03-3.lifeway.org/172.17.20.33:2181. Will not attempt to authenticate using SASL (unknown error) (ClientCnxn:1019) 2018-12-11 00:12:48,690 ERROR - [pool-2-thread-7 - 3b35712a-6c84-4102-a19c-cedf26ea5e1c:] ~ graph rollback due to exception AtlasBaseException:Instance __AtlasUserProfile with unique attribute {name=admin} does not exist (GraphTransactionInterceptor:73) Let me know how we can see lineage of Managed and virtual view tables. ... View more

@Ronak bansal We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 We are getting below error while running import-hive.sh script. root@hn:/var/log/atlas# tail -f application.log 2018-12-10 04:50:06,783 ERROR - [pool-2-thread-7 - d35c5859-a93e-40ce-94ff-5c12e45f2880:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_awood@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-10 04:50:07,100 ERROR - [pool-2-thread-7 - 0b130511-ed2a-4434-95ad-692b0babf348:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_type2@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-10 04:50:07,431 ERROR - [pool-2-thread-7 - fbf49dd4-12fd-4ff1-bdf7-d69187ead26c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_type2_new@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-10 04:50:07,826 ERROR - [pool-2-thread-7 - 1041077f-a6cf-4b24-8669-d84d88155d82:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-10 04:50:08,056 ERROR - [pool-2-thread-9 - ce6d8709-db1a-458e-8cfb-4d1ccbb28ca6:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2_cdc@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-10 04:50:08,332 ERROR - [pool-2-thread-7 - 1757d3a0-4562-4c9b-bf1b-526cd34b2c05:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2_current@C03} does not exist (GraphTransactionInterceptor:73) In Atlas we are able to see External tables linage but not for Managed and Virtual Views tables. Could you please let me why we are getting this error. Can we see linage for Managed and Virtual view tables? Thanks, Owez ... View more

@Sharmadha Sainath @Ana Gillan Hi Team, We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 Questions: 1) We need to know can we see linage of Managed tables and virtual views linage in Atlas? 2) When we create a new tables and try to see linage of that tables in first attempt we are not able to see linage of that tables after running import-hive.sh script we are able to see the linage of that newly created tables, why this is happening? it should show automatically the linage of newly created tables. 3) Could you please let me know is there any property need to set or any configuration needs to change. Two snap shot has been attached 1) atlas_external table as we can see linage of that table after running import-hive.sh script. 2) atlas_managed table we cannot see linage of that table even after running script. Request you to let me know that can we see linage for external tables only? or we can see linage for managed tables and virtual views as well? There is no error in application.log, hbase-log, ranger-audit, kafka-log. Permission has been set in ranger. Please suggest. We are stuck now. Thanks, Owez ... View more

Hi Team, We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 Now we are able to see linage of older tables as well, but only for External tables. As when we are trying to create a new managed tables and when we create a joins/intersecting two tables, we are not able to see their linage. if we create a new tables as external tables whereas we can see the linage of that tables after running import-hive.sh script. Request you to let me know that can we see linage for external tables only? or we can see linage for managed tables as well? @team Please let me know what we can do in Atlas and what we cannot do in Atlas. ... View more