Lineage data failed to fetch in Atlas Web UI for large tables

[#1]

On my atlas web UI, selecting a table takes too much time to fetch lineage and incase of big tables after loading for too long it just displays "No Lineage Data Found" on checking the requests made which are taking long I found one REST API's being called 2 times which take somewhere between 3-5 mins. Sometimes it successfully fetches data and displays lineage but in case of too big tables no lineage data appears.

The API which is being called twice is:-

Request URL:http://hostname:port/api/atlas/lineage/c797bffa-37d0-4b23-b168-998ffc4855e3/inputs/graph?_=1487780877854

JSON Response

{"requestId":"qtp1177377518-69414 - 3e3301e8-4944-429e-96fe-9ef640684372","results":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"__tempQueryResultStruct523","values":{"vertices":{},"edges":{}}}} 

Time Taken 3.7 minutes.

I guess the request is taking too long to process and hence empty vertices and edges are recieved in the JSON Response. Is there someway I can increase the time for this request to be processed or better improve the performance altogether by using some caching mechanism or something.

[#2]

@Poorvi Sachar

What does the atlas application log say, when this request is made? which version of HDP is this? Is it possible to attach the application log here for debugging?

[#3]

As a am a newbie could you please tell me which logs to attach here..

[#4]

Atlas logs are stored in application.log file which can be found at <ATLAS_HOME>/logs/application.log.

[#5]

@Poorvi Sachar Are you still seeing this issue? if yes, please attach the application logs?

[#6]

@Ayub Khan

Yes for few tables the lineage result shows as - "Lineage data not found". I am not sure that whether it has nothing to show or it just fetches null edges and vertices because the api call takes a longer duration to complete.

Do you need to see some specific section of the application logs? Also I did not find anything related to the issue but incase you still need it then maybe you can tell me what exactly are you looking for as the logs contain some extra information.

[#7 · Accepted solution]

@Poorvi Sachar

Same API is not called twice . It must be

http://hostname:port/api/atlas/lineage/c797bffa-37d0-4b23-b168-998ffc4855e3/inputs/graph

and

http://hostname:port/api/atlas/lineage/c797bffa-37d0-4b23-b168-998ffc4855e3/outputs/graph

Can you please check ? If it's not , it is a bug possibly.

[#8]

yeah sorry its two different apis...could still you please help me figure out why lineage data is not being fetched or how could i really check in the db it there is no data....also is there some method to improve this performance...as it usually takes 3-4 mins to fetch data for small tables which is way too much

[#9]

How long does it take for other APIs ? Atlas server might be residing in a remote location and you might be pinging from your local machine hence there can be network delay. Can you please login to the machine and do curl GET of the same APIs and add -w to curl to check the time taken ?

Also,

>> I guess the request is taking too long to process and hence empty vertices and edges are recieved in the JSON Response.

This is not true. UI says "Lineage not found" when there is really no lineage data. Truthfulness of this API response doesn't depend on the time taken. We should debug why the API call takes long time.

[#10]

@Poorvi Sachar

If it shows "Lineage not found" , it is not an exception or error . It means that actually there is no Lineage.