question Re: Lineage data failed to fetch in Atlas Web UI for large tables in Support Questions

Lineage data failed to fetch in Atlas Web UI for large tables

poorvi_sachar — Thu, 23 Feb 2017 01:55:36 GMT

On my atlas web UI, selecting a table takes too much time to fetch lineage and incase of big tables after loading for too long it just displays "No Lineage Data Found" on checking the requests made which are taking long I found one REST API's being called 2 times which take somewhere between 3-5 mins. Sometimes it successfully fetches data and displays lineage but in case of too big tables no lineage data appears.

The API which is being called twice is:-

Request URL:http://hostname:port/api/atlas/lineage/c797bffa-37d0-4b23-b168-998ffc4855e3/inputs/graph?_=1487780877854

JSON Response

{"requestId":"qtp1177377518-69414 - 3e3301e8-4944-429e-96fe-9ef640684372","results":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct523","values":{"vertices":{},"edges":{}}}}

Time Taken 3.7 minutes.

I guess the request is taking too long to process and hence empty vertices and edges are recieved in the JSON Response. Is there someway I can increase the time for this request to be processed or better improve the performance altogether by using some caching mechanism or something.

Re: Lineage data failed to fetch in Atlas Web UI for large tables

apathan — Thu, 23 Feb 2017 11:04:39 GMT

@Poorvi Sachar

What does the atlas application log say, when this request is made? which version of HDP is this? Is it possible to attach the application log here for debugging?

Re: Lineage data failed to fetch in Atlas Web UI for large tables

ssainath — Thu, 23 Feb 2017 20:02:14 GMT

@Poorvi Sachar

Same API is not called twice . It must be

http://hostname:port/api/atlas/lineage/c797bffa-37d0-4b23-b168-998ffc4855e3/inputs/graph

and

http://hostname:port/api/atlas/lineage/c797bffa-37d0-4b23-b168-998ffc4855e3/outputs/graph

Can you please check ? If it's not , it is a bug possibly.

Re: Lineage data failed to fetch in Atlas Web UI for large tables

poorvi_sachar — Fri, 24 Feb 2017 11:53:44 GMT

As a am a newbie could you please tell me which logs to attach here..

Re: Lineage data failed to fetch in Atlas Web UI for large tables

poorvi_sachar — Fri, 24 Feb 2017 11:55:27 GMT

yeah sorry its two different apis...could still you please help me figure out why lineage data is not being fetched or how could i really check in the db it there is no data....also is there some method to improve this performance...as it usually takes 3-4 mins to fetch data for small tables which is way too much

Re: Lineage data failed to fetch in Atlas Web UI for large tables

apathan — Fri, 24 Feb 2017 13:31:35 GMT

Atlas logs are stored in application.log file which can be found at <ATLAS_HOME>/logs/application.log.

Re: Lineage data failed to fetch in Atlas Web UI for large tables

apathan — Mon, 27 Feb 2017 16:38:58 GMT

@Poorvi Sachar Are you still seeing this issue? if yes, please attach the application logs?

Re: Lineage data failed to fetch in Atlas Web UI for large tables

ssainath — Tue, 28 Feb 2017 17:00:48 GMT

How long does it take for other APIs ? Atlas server might be residing in a remote location and you might be pinging from your local machine hence there can be network delay. Can you please login to the machine and do curl GET of the same APIs and add -w to curl to check the time taken ?

Also,

>> I guess the request is taking too long to process and hence empty vertices and edges are recieved in the JSON Response.

This is not true. UI says "Lineage not found" when there is really no lineage data. Truthfulness of this API response doesn't depend on the time taken. We should debug why the API call takes long time.

Re: Lineage data failed to fetch in Atlas Web UI for large tables

poorvi_sachar — Wed, 01 Mar 2017 22:26:05 GMT

@Ayub Khan

Yes for few tables the lineage result shows as - "Lineage data not found". I am not sure that whether it has nothing to show or it just fetches null edges and vertices because the api call takes a longer duration to complete.

Do you need to see some specific section of the application logs? Also I did not find anything related to the issue but incase you still need it then maybe you can tell me what exactly are you looking for as the logs contain some extra information.

Re: Lineage data failed to fetch in Atlas Web UI for large tables

ssainath — Sat, 04 Mar 2017 03:39:01 GMT

@Poorvi Sachar

If it shows "Lineage not found" , it is not an exception or error . It means that actually there is no Lineage.