@Owez Mujawar Could you run kafka console consumer on topic ATLAS_HOOK and ATLAS_ENTITIES when you create a table and check if the messages are flowing to the topic ? ... View more

@Owez Mujawar What is the offsets.topic.replication.factor in kafka properties ? Is it greater than the number of kafka brokers ? Can you check kafka logs for error messages like the following ? ERROR [KafkaApi-1001] Number of alive brokers 'x' does not meet the required replication factor 'x' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis) ... View more

@Owez Mujawar Looks like Hive Hook messages are not read by Atlas. Can you check if Hive hook is enabled correctly and can you check the following permissions ? hive's permission to publish to ATLAS_HOOK topic atlas's permission to read from ATLAS_HOOK topic and publish to ATLAS_ENTITIES topic. ... View more

@Owez Mujawar It is not possible to see lineage of tables created before Atlas Hive hook is enabled. Once hive hook is enabled , and then when new tables,lineage are created , you will be able to see the lineage. Let's consider the scenario: 1. Atlas is not installed on the cluster and no hive hook is enabled . Tables , views , (table1,view1)) etc are created in Hive. 2. Install Atlas on the cluster . Run import-hive.sh to let the script contact Hive metastore and populate Atlas with existing Hive data. This script creates all tables , but lineage is not created (except for the external table). 3. Now , Enable hive hook .Create tables and views now . Since Hive hook is enabled , the lineage should be created in Atlas without running any import script. Since hook is enabled , we don't need to run the import-hive.sh script anymore. ... View more

@Owez Mujawar Once you enable Hive hook , you should see the table, lineage , all DDL changes of Hive in Atlas. Please refer http://atlas.apache.org/Hook-Hive.html to enable hive hook. import-hive.sh script is a one time script (no harm running multiple times though), which is used when on-boarding Atlas for the first time to sync metadata of Hive with Atlas. Import-hive.sh script basically contacts Hive metastore and populates Hive entities and sends it to Atlas. The script has no way to know how a CTAS table is created and what is the source of the table , hence lineage cannot be created. But it can know the details of external table. Hook captures live events and sends the information to Atlas. ... View more

@Owez Mujawar Import-hive.sh doesn't capture the lineage for internal tables. When Hive hook is enabled , and when you execute the queries, you can see the lineage in Atlas. ... View more

@Takefumi Oide The tag associated to the deleted entity can't be disassociated and hence can't be deleted. ... View more