@Sunile Manjee
Atlas creates two topics Atlas_Entities and Atlas_Hook. When a Hook fires it will send all of the meta data passed to it into Atlas entities and send them to Atlas via the Atlas_Hook topic. When Atlas successfully creates the new entities it received from aHook, it will publish the resulting entities to the Atlas_Entities topic. You can watch either topic to know that an entity or set of entities are being created or a request to create them has been sent. You can also go back and read the entire topic from the first available offset to see what entities or sets of entities have been created over that period. You can then calculate lineage using the same graph processing techniques used by Titan (the Graph API used by Atlas). However, there is no actual lineage information actually on the topic, just the JSON that describes the entities being created and references to other entities. This is because Kafka is nothing more than a message bus, it buffers messages for asynchronous read. It cannot do Graph calculation and even if it could, it only retains data for a limited period of time. Thus Atlas uses Titan to calculate lineage based on data stored in Hbase.
