atlas hive metadata

[#1]

We have HDP 2.6, when i added table & column comments using alter table.These comments are not reflected in Atlas.

I thought this should be automated

[#2]

@vishal dutt

Can you elaborate on what are you intend to have an output?

Did you enable the Ranger plugins for Atlas and hive ?

[#3]

I want to see the table & column comments in Atlas hive table schema table. Yes ranger plugins are enabled for hive & atlas

[#4]

@vishal dutt

You could be facing this issue

https://community.hortonworks.com/answers/31485/view.html

Never the less I am trying it out, what I am sure of is CTAS works. Can you exactly what you did and see if I can reproduce it

[#5]

I have a external table already created in Hive, i run the alter table to add comment.

alter table tblstat_tmp change id id int COMMENT 'Unique ID';

[#6]

It seems , i have run the script ./usr/hdp/current/atlas-server/hook-bin/import-hive.sh

[#7]

Could you also make sure your hive-site has Atlas Hive Hook in hive.exec.post.hooks from Ambari?

[#8]

@vishal dutt

I have reproduced successfully your issue

• HDP 2.6.2.0 Non-Kerberized
  
• Atlas 0.8.0
  
• Ranger 0.7.0

I first enable the ranger plugins for all for some eventual testing. I think you only need to run the import-hive.sh for the objects created before enabling the plugins

I will skip the creation of the hdfs , copying of the data and creation of the table.

Created a database

hive> create database jair; 

Validate the creation of DB

hive> show databases; 
OK 
default 
jair 
Time taken: 0.1 seconds, Fetched: 2 row(s) 

Switched to my DB

hive> use jair; 
OK Time taken: 0.282 seconds 

Checked the available table

hive> show tables; 
OK tmp

Switched user database and run CTAS

hive> create table toto  as select * from tmp; 
Query ID = sheltong_20180407145450_a12e7cf8-27f5-4422-812a-02ceb8cd8065
Total jobs = 1
Launching Job 1 out of 1
Tez session was closed. Reopening...
Session re-established.
Status: Running (Executing on YARN cluster with App id application_1523096322530_0002)
--------------------------------------------------------------------------------
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
--------------------------------------------------------------------------------
Map 1 ..........   SUCCEEDED      1          1        0        0       0       0
--------------------------------------------------------------------------------
VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 41.27 s
--------------------------------------------------------------------------------
Moving data to directory hdfs://nakuru.kenya.ke:8020/apps/hive/warehouse/jair.db/toto
Table jair.toto stats: [numFiles=1, numRows=14, totalSize=196, rawDataSize=182]
OK
Time taken: 85.602 seconds 

Added comment see attached screenshot comment_captured.jpg

hive> alter table tmp change a a string COMMENT 'show me'; 
OK Time taken: 8.11 seconds 

Added a new comment

After the refresh I see the comment now see attached screenshot comment_dutt.jpg

hive> alter table tmp change b b string COMMENT 'Answer by vishal dutt'; 
OK Time taken: 0.813 seconds

I think your problem would be in the navigation.

In the Atlas UI select, Basic search and the search type should be hive_table on the right pane you should see your table click on the desired table, you should be able to see the lineage and below in the details Properties, Tags, Audits chose Schema and you should be able to see your comment. See attached screenshots !!!

Please let me know if it worked for you.

[#9]

@vishal dutt - Validate your hive section : Advanced hive-atlas-application.properties in Ambari

• atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false. Recommended to be set to false to avoid delays in hive query completion.

[#10]

@vishal dutt

I updated this thread 3 days ago with a detailed process and screenshots did you go through that?

If so revert with some input.

[#11]

Thanks Geoffrey. It works after i have run the script ./usr/hdp/current/atlas-server/hook-bin/import-hive.sh

[#12]

@vishal dutt

You ONLY run import-hive.sh for table/objects created before enabling the Ranger-hive plugin, after enabling the plugin the hook should automate that process.

Could you validate by testing now with the plugin enabled and update and close the thread too.And if you feel my answer helped "Accept"
Thank you

[#13]

Ranger plugins for Atlas and hive are enabled. Automatic sync will happen only if Advanced hive-atlas-application.properties is true in Ambari atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false. Recommended to be set to false to avoid delays in hive query completion.