How Atlas know that Table is created in Hive

[#1]

I know that Atlas did it automatically when we make necessary configurations for import-hive.sh etc files. But I want to know internal working of Atlas. What happens behind the scene. Which class/module is invoked/involved?

[#2 · Accepted solution]

@Muhammad Imran Tariq,

Atlas Hive Hook in configured in Hive by set-up in hive-site.xml.

   <property>
      <name>hive.exec.post.hooks</name>
      <value>org.apache.atlas.hive.hook.HiveHook</value>
    </property>

whenever a table is created in hive, a event is triggered which invokes the atlas hive hook. This hook sends message to Atlas via notification to kafka, The notification in kafka is consumed by Atlas and hive entity is created.

Refer:-

http://atlas.incubator.apache.org/Bridge-Hive.html

[#3]

cool. So what type of information Hook gets from Hive and how? I basically wants to know that. Did Hook query Hive to know information?

[#4]

@Muhammad Imran Tariq

Following information is passed from Hive to Atlas-Hive Hook from Hive whenever any DML statement is executed when Atlas Hive hook is configured.

refer this code on github -

final HiveEventContext event = new HiveEventContext();
            event.setInputs(hookContext.getInputs());
            event.setOutputs(hookContext.getOutputs());
            event.setHookType(hookContext.getHookType());
            final UserGroupInformation ugi = hookContext.getUgi() == null ? Utils.getUGI() : hookContext.getUgi();
            event.setUgi(ugi);
            event.setUser(getUser(hookContext.getUserName(), hookContext.getUgi()));
            event.setOperation(OPERATION_MAP.get(hookContext.getOperationName()));
            event.setQueryId(hookContext.getQueryPlan().getQueryId());
            event.setQueryStr(hookContext.getQueryPlan().getQueryStr());
            event.setQueryStartTime(hookContext.getQueryPlan().getQueryStartTime());
            event.setQueryType(hookContext.getQueryPlan().getQueryPlan().getQueryType());
            event.setLineageInfo(hookContext.getLinfo());

[#5]

This means Hive will automatically notify Atlas. What if I have MySQL instead of Hive. I am sure MySQL will not notify like Hive. Right? If yes then what can be done to achieve notification in case of MySQL?

[#6]

@Muhammad Imran Tariq, You are right MySQL will not notify like Hive, check if MySql has any mechanism to invoke the class / method when there is a change in metadata.

Do share your findings

Nixon

[#7]

https://dev.mysql.com/doc/refman/5.7/en/writing-plugins.html

MySQL does offer capability to write plugins which might offer similar functionality as the Hive|AtlasHook.