question Re: Select clause is not working in Atlas DSL in Archives of Support Questions (Read Only)

Select clause is not working in Atlas DSL

saba_baig — Mon, 10 Jul 2017 19:46:37 GMT

Hi,

Can anyone please help me in understanding SELECT clause in DSL? Following mentioned query returns no rows though there is an entity named as "Reporting" of type "DB" having attributes as name and owner.

DB where name="Reporting" select name, owner

Re: Select clause is not working in Atlas DSL

ssainath — Mon, 10 Jul 2017 20:05:58 GMT

@Saba Baig

The select lets you "select" a particular attribute of an entity

DB where name="Reporting" select name, owner

displays the name and owner of db whose name is "Reporting"

Currently UI doesn't display results but REST GET API call actually returns the result.

Example :

DSL query : hive_table name = "table1" select qualifiedName,owner

response :

{queryType: "DSL",
queryText: "`hive_table` name="table1" select qualifiedName,owner",
attributes: 
{name: 
["qualifiedName",
"owner"
],
values: 
[
["default.table1@cl1",
"atlas"
]
]
}
}

Re: Select clause is not working in Atlas DSL

saba_baig — Tue, 11 Jul 2017 13:40:37 GMT

@Sharmadha Sainath can you please guide me on this? as the method "searchUsingDslQuery" from EntityDiscoveryService.java is getting the result of the fetched attributes but then it only adds the attributes to the object of AtlasSearchResult...neither an AtlasVertex nor contains theInstance or theTrait.... Could you please guide me on this?

Re: Select clause is not working in Atlas DSL

saba_baig — Tue, 11 Jul 2017 13:51:02 GMT

@Sharmadha Sainath Could you please also guide me how is it using groupby and other functions liek max, min with some example? Does their result reflects on UI or not? Also, what features e.g wildcards (? . * etc) are available in full-text provided by Atlas? and how can we use them (examples)?

Re: Select clause is not working in Atlas DSL

saba_baig — Sun, 18 Aug 2019 00:18:36 GMT

Queries like this are also not returning anything e.g using groupby with select

Re: Select clause is not working in Atlas DSL

saba_baig — Tue, 11 Jul 2017 14:27:28 GMT

@Sharmadha Sainath could you please also tell me how does this works

`Log Data` ??? what am I supposed to write in single quotes and how do you explain the results?

Re: Select clause is not working in Atlas DSL

ssubramanian — Tue, 11 Jul 2017 15:14:12 GMT

@Saba Baig, you can search for any entities tagged with a particular tag name using the single quotes. for e.g. `Log Data` searches for all entities in atlas tagged as Log Data. Only classification name is supported in single quotes.

Re: Select clause is not working in Atlas DSL

ssainath — Tue, 11 Jul 2017 15:20:46 GMT

@Saba BaigFew aggregate functions are not working in V2 APIs.

There are few Apache JIRAs filed for the same :

https://issues.apache.org/jira/browse/ATLAS-1874

https://issues.apache.org/jira/browse/ATLAS-1639

However V1 APIs work perfectly .

For example :

Query : hive_table groupby(owner) select owner,count()

URL encoded :

http://localhost:21000/api/atlas/discovery/search/dsl?limit=25&query=hive_table+groupby(owner)+select+owner%2Ccount() 
Response :

{requestId: "pool-2-thread-9 - 8a45b7a6-0cd9-4e9d-8b71-66e313f5abf1",
query: "hive_table groupby(owner) select owner,count()",
queryType: "dsl",
count: 2,
results: 
[
{$typeName$: "__tempQueryResultStruct74",
count(): 9,
owner: "hrt_qa"
},

{$typeName$: "__tempQueryResultStruct74",
count(): 2,
owner: "anonymous"
}
],
dataType: 
{typeName: "__tempQueryResultStruct74",
typeDescription: null,
typeVersion: "1.0",
attributeDefinitions: 
[
{name: "owner",
dataTypeName: "string",
multiplicity: 
{lower: 0,
upper: 1,
isUnique: false
},
isComposite: false,
isUnique: false,
isIndexable: false,
reverseAttributeName: null
},

{name: "count()",
dataTypeName: "long",
multiplicity: 
{lower: 0,
upper: 1,
isUnique: false
},
isComposite: false,
isUnique: false,
isIndexable: false,
reverseAttributeName: null
}
]
}
}

This query groups the owner of hive_table and and lists the owner ,count in each group . The concept is similar to that of SQL.

Since the UI currently uses V2 query , you may not see the results on UI. But you can always rely on REST APIs.

For this particular kind of queries , please use V1 APIs for now .

Re: Select clause is not working in Atlas DSL

ssainath — Tue, 11 Jul 2017 15:30:40 GMT

@Saba Baig

>> Also, what features e.g wildcards (? . * etc) are available in full-text provided by Atlas? and how can we use them (examples)?

Atlas uses lucene query syntax for its full text search (basic search).

Please refer lucene query usage here :

http://lucene.apache.org/core/4_1_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#Wildcard_Searches

Example:

If I have 3 hive_table entities in Atlas , table_a , table_b , table_ab

Query :
1)
type = hive_table , query = table_?

http://localhost:21000/api/atlas/v2/search/basic?limit=25&excludeDeletedEntities=true&query=table_?&typeName=hive_table

Returns 2 entities (table_a , table_b)

2)
type = hive_table , query = table_*

http://localhost:21000/api/atlas/v2/search/basic?limit=25&excludeDeletedEntities=true&query=table_*&typeName=hive_table

Returns 3 entities (table_a , table_b , table_ab)

I hope this example makes it clear ...

Note : Wild card search can be done on UI itself.

Re: Select clause is not working in Atlas DSL

saba_baig — Sun, 18 Aug 2019 00:18:28 GMT

@Sharmadha Sainath

I have an entity of type "Table" named as "sales_fact". I tried to get that using api

http://localhost:21000/api/atlas/v2/search/basic?limit=25&includeDeletedEntities=true&query=sales_fa?t&typeName=Table

=====Response =====
{  "queryType": "BASIC",  "type": "Table",  "entities": [  {  "typeName": "Table",  "attributes": {  "owner": "Joe",  "qualifiedName": "sales_fact",  "name": "sales_fact",  "description": "sales fact table"  },  "guid": "c9838518-041f-4759-b3e5-01147b1526d1",  "status": "ACTIVE",  "displayText": "sales_fact",  "classificationNames": [  "Fact"  ]  }  ]}

But when I tried to the same search (Basic using query) I typed sales_fa?t and I got invalid Expression. I added a breakpoint in the API but it was never called. I think there is some validation applied on that field that omits any character like ? from the query before submitting it to API. Please guide.

I think, this could be the reason

Apache Lucene Analysis Pipeline

Re: Select clause is not working in Atlas DSL

ssainath — Tue, 11 Jul 2017 17:39:56 GMT

@Saba Baig ,

>> But when I tried to the same search (Basic using query) I typed sales_fa?t and I got invalid Expression

What is the query you fired ?

You mentioned http://localhost:21000/api/atlas/v2/search/basic?limit=25∈cludeDeletedEntities=true&query=sales_fa?t&typeName=Table works for you. I am not able to get your question.

Re: Select clause is not working in Atlas DSL

saba_baig — Sun, 18 Aug 2019 00:18:20 GMT

@Sharmadha Sainath

You also mentioned that if we write anything in single quotes it gives you all the entities with that tag. I found this behaving the same in case of DSL (Advanced Search)

Search for `Fact` in Basic Search

But, searching for `Fact` in Basic (Search by Query) behaved differently...

Question 1: If we follow what you just said then why am I seeing Log Data and Metric appearing there?

Question 2: Secondly, you said "Only classification name is supported in single quotes." then,

2.1) What is a classification name?

2.2) Difference between tag and classification name?

2.3) What to mention in single quotes... Classification name or tag?

Re: Select clause is not working in Atlas DSL

saba_baig — Tue, 11 Jul 2017 18:19:15 GMT

@Sharmadha Sainath I said that I got invalid expression when fired the same query in the Atlas UI for basic search using query. Here it gave me Invalid Expression for the same query (sales_fa?t) that worked via api as mentioned earlier.

Re: Select clause is not working in Atlas DSL

ssainath — Tue, 11 Jul 2017 19:04:26 GMT

2.1) What is a classification name?

2.2) Difference between tag and classification name?

Classification and tags are same

Basic search is basically Lucene search . So ,If you see your result in the screenshot, basic search has returned everything related to fact.(i.e) entities with description containing fact.

Re: Select clause is not working in Atlas DSL

ssainath — Tue, 11 Jul 2017 19:06:14 GMT

@Saba Baig

Please refer to this issue

https://issues.apache.org/jira/browse/ATLAS-1788

Re: Select clause is not working in Atlas DSL

saba_baig — Tue, 11 Jul 2017 19:18:03 GMT

Thanks @Sharmadha Sainath