https://community.cloudera.com/t5/Support-Questions/How-Atlas-is-notified-when-there-is-a-change-in-any-data/m-p/209623#M171577 <P><A rel="user" href="https://community.cloudera.com/users/17468/sababaig.html" nodeid="17468">@Saba Baig</A> </P><P> &gt;&gt; How Hive notifies Atlas about any DML/DDL operation in Atlas against which Atlas generates lineage?</P><P>Whenever there is any metadata change events in Hive , HiveHook captures it and puts the details of created/updated hive entity to a kafka topic called ATLAS_HOOK. Atlas is the consumer of the ATLAS_HOOK. So Atlas gets the message from ATLAS_HOOK.</P><P>&gt;&gt; what is the information that Hive sends to Atlas?</P><P>Example :</P><PRE>hive &gt; create table emp(id int,name string);</PRE><P>1.HiveHook composes a JSON message that contains information about table name , database , columns and other table properties and sends it to ATLAS_HOOK.</P><P>2. ATLAS_HOOK queues up the messages from HiveHook and Atlas consumes from it. Atlas consumes the JSON message about table emp and ingests it.</P><PRE>hive &gt; create table t_emp as select * from emp;</PRE><P>1.HiveHook composes JSON message that contains t_emp details and also the source table name (emp) and sends to ATLAS_HOOK. </P><P>2.Atlas understands from the JSON message consumed from ATLAS_HOOK , that it is a CTAS table and it has a source table , ingests the table t_emp and constructs lineage for the tables emp and t_emp.</P><P>&gt;&gt; is Hive DB going to notify Atlas Server on its own or HiveHook is going to check constantly in the Hive DB and pull the changes</P><P>HiveHook doesn't check hive constantly all time. Whenever there is any metadata event change ( like when user fires a hive query that involves creation/updation/drop ) , HiveHook notifies ATLAS_HOOK.</P><P>NOTE :</P><P>If you want to know more about the exact JSON content sent by HiveHook , you can create a table in hive and check the message that lands in ATLAS_HOOK for that table.</P> Thu, 15 Jun 2017 15:41:34 GMT ssainath 2017-06-15T15:41:34Z