Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

atlas hive metadata

atlas hive metadata

New Contributor

We have HDP 2.6, when i added table & column comments using alter table.These comments are not reflected in Atlas.

I thought this should be automated

12 REPLIES 12
Highlighted

Re: atlas hive metadata

Mentor

@vishal dutt

Can you elaborate on what are you intend to have an output?

Did you enable the Ranger plugins for Atlas and hive ?

Re: atlas hive metadata

New Contributor

I want to see the table & column comments in Atlas hive table schema table. Yes ranger plugins are enabled for hive & atlas

Re: atlas hive metadata

Mentor

@vishal dutt

You could be facing this issue

https://community.hortonworks.com/answers/31485/view.html

Never the less I am trying it out, what I am sure of is CTAS works. Can you exactly what you did and see if I can reproduce it

Re: atlas hive metadata

New Contributor

I have a external table already created in Hive, i run the alter table to add comment.

alter table tblstat_tmp change id id int COMMENT 'Unique ID';

Re: atlas hive metadata

New Contributor

It seems , i have run the script ./usr/hdp/current/atlas-server/hook-bin/import-hive.sh

Re: atlas hive metadata

Could you also make sure your hive-site has Atlas Hive Hook in hive.exec.post.hooks from Ambari?

Re: atlas hive metadata

Mentor

@vishal dutt

I have reproduced successfully your issue

  • HDP 2.6.2.0 Non-Kerberized
  • Atlas 0.8.0
  • Ranger 0.7.0

I first enable the ranger plugins for all for some eventual testing. I think you only need to run the import-hive.sh for the objects created before enabling the plugins

I will skip the creation of the hdfs , copying of the data and creation of the table.

Created a database

hive> create database jair; 

Validate the creation of DB

hive> show databases; 
OK 
default 
jair 
Time taken: 0.1 seconds, Fetched: 2 row(s) 

Switched to my DB

hive> use jair; 
OK Time taken: 0.282 seconds 

Checked the available table

hive> show tables; 
OK tmp

Switched user database and run CTAS

hive> create table toto  as select * from tmp; 
Query ID = sheltong_20180407145450_a12e7cf8-27f5-4422-812a-02ceb8cd8065
Total jobs = 1
Launching Job 1 out of 1
Tez session was closed. Reopening...
Session re-established.
Status: Running (Executing on YARN cluster with App id application_1523096322530_0002)
--------------------------------------------------------------------------------
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
--------------------------------------------------------------------------------
Map 1 ..........   SUCCEEDED      1          1        0        0       0       0
--------------------------------------------------------------------------------
VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 41.27 s
--------------------------------------------------------------------------------
Moving data to directory hdfs://nakuru.kenya.ke:8020/apps/hive/warehouse/jair.db/toto
Table jair.toto stats: [numFiles=1, numRows=14, totalSize=196, rawDataSize=182]
OK
Time taken: 85.602 seconds 

Added comment see attached screenshot comment_captured.jpg

hive> alter table tmp change a a string COMMENT 'show me'; 
OK Time taken: 8.11 seconds 

Added a new comment

After the refresh I see the comment now see attached screenshot comment_dutt.jpg

hive> alter table tmp change b b string COMMENT 'Answer by vishal dutt'; 
OK Time taken: 0.813 seconds

I think your problem would be in the navigation.

In the Atlas UI select, Basic search and the search type should be hive_table on the right pane you should see your table click on the desired table, you should be able to see the lineage and below in the details Properties, Tags, Audits chose Schema and you should be able to see your comment. See attached screenshots !!!

Please let me know if it worked for you.


comment-dutt.jpgcomment-captured.jpg

Re: atlas hive metadata

Rising Star

@vishal dutt - Validate your hive section : Advanced hive-atlas-application.properties in Ambari

  • atlas.hook.hive.synchronous - boolean, true to run the hook synchronously. default false. Recommended to be set to false to avoid delays in hive query completion.

Re: atlas hive metadata

Mentor

@vishal dutt

I updated this thread 3 days ago with a detailed process and screenshots did you go through that?

If so revert with some input.