question Re: Atlas hive is not showing lineage data in Archives of Support Questions (Read Only)

Atlas hive is not showing lineage data

Manus — Fri, 24 Jun 2016 17:04:59 GMT

Hi,

I have created tables for lineage from https://community.hortonworks.com/content/kbentry/36121/using-apache-atlas-to-view-data-lineage.html

I am able to see tables on apache atlas gui..

But I am not able to see lineage data..

lineage.png

Re: Atlas hive is not showing lineage data

Manus — Fri, 24 Jun 2016 19:56:14 GMT

Hello All,

Right now,I have installed Apache atlas (0.6 version) on HDP-2.4.0.My first question is will Apache atlas work fine on HDP-2.4.0?.As per my understanding Atlas 0.6 version cant be installed on HDP-2.4.0 but somehow,I have installed it on my local machine.

Currently,I am able to see the metadata about hive tables but for that, I have to run import-hive.sh repeatedly which is not expected behavior in Apache Atlas.How can we avoid running import-hive.sh repeatedly?.

and also not getting lineage for hive tables in Atlas?.

I am sure that I have configured atlas correctly for getting metadata from hive,seeing lineage and almost tried all the solutions from we sites but not able to solve the problem.

The links which I have tried for configuration:

1) http://atlas.incubator.apache.org/Configuration.html

2) https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/configuring_atlas.html

please help me,

Re: Atlas hive is not showing lineage data

vvaks — Mon, 04 Jul 2016 10:59:08 GMT

@Manoj Dhake

Download and use Sandbox 2.5, it was just released and is running Atlas 0.7.

https://hortonworks.com/downloads/#tech-preview

This version of Atlas is much better than 0.5 and 0.6 and should come pre-installed. Also, remember that in order to see lineage for an entity, you need to run a process that creates a new entity or modifies the existing entity. The simplest example is running a create table ... select ... from... SQL statement.

Re: Atlas hive is not showing lineage data

wnascimento — Fri, 15 Jul 2016 00:23:17 GMT

Hello @Vadim

I tried creaing a new entity via create table as select... and now I can see the lineage, but for an existing entity, what do I have to do to see the lineage?

Thanks

Re: Atlas hive is not showing lineage data

mjohnson — Fri, 15 Jul 2016 01:09:37 GMT

@Washington Nascimento

Check out the section Importing Hive Metadata from the link: http://atlas.incubator.apache.org/Bridge-Hive.html. You will want to utilize the import-hive.sh script to perform the initial load of metadata.

Re: Atlas hive is not showing lineage data

wnascimento — Fri, 15 Jul 2016 01:42:02 GMT

Great @mjohnson!

Thank you !

Re: Atlas hive is not showing lineage data

RussHockey2 — Tue, 19 Jul 2016 19:50:57 GMT

Hello Vadim,

Thank for identifying thew newer release of Atlas 0.7

I can see that you folks have been very busy creating new features - especially in the area of a Taxonomy. Great Job !!!

So I have written java code which will be highlighting our (IBM's) ability to pull metadata from various sources, and this newer release seems primed for working with your release.

A few observations :

1) While the Apache Atlas interface recognizes newly added entities - the lineage display seems broken or disabled. It only shows a flat line. Any ideas why?

2) Regardless of the Lineage display - my code to call the API with say:

http://192.168.1.133:21000/api/atlas/lineage/hive/table/default.anewsample@Sandbox/inputs/graph

returns me the expected JSON structure of edges/ and verticies

and from each item in the edge i can further query details of the entity using :

http://192.168.1.133:21000/api/atlas/entities/>

Question: How do I now what is the correct order of the lineage? Bear in mind, that JSON returned results can never assure a given order. Any help to understanding the order problem?

Regards,

Russ : rga@us.ibm.com

Re: Atlas hive is not showing lineage data

vvaks — Wed, 27 Jul 2016 02:06:56 GMT

@Russell AndersonSorry it took a while to respond, I have not been back to this question ina while. Could you ask the questions in a separate thread so that the answer is easier to find for others? I will do my best to answer.