Select clause is not working in Atlas DSL

[#1]

Hi,

Can anyone please help me in understanding SELECT clause in DSL? Following mentioned query returns no rows though there is an entity named as "Reporting" of type "DB" having attributes as name and owner.

DB where name="Reporting" select name, owner

[#2 · Accepted solution]

@Saba Baig

The select lets you "select" a particular attribute of an entity

DB where name="Reporting" select name, owner

displays the name and owner of db whose name is "Reporting"

Currently UI doesn't display results but REST GET API call actually returns the result.

Example :

DSL query : hive_table name = "table1" select qualifiedName,owner

response :

{queryType: "DSL",
queryText: "`hive_table` name="table1" select qualifiedName,owner",
attributes: 
{name: 
["qualifiedName",
"owner"
],
values: 
[
["default.table1@cl1",
"atlas"
]
]
}
}

[#3]

@Sharmadha Sainath Could you please also guide me how is it using groupby and other functions liek max, min with some example? Does their result reflects on UI or not? Also, what features e.g wildcards (? . * etc) are available in full-text provided by Atlas? and how can we use them (examples)?

[#4]

Queries like this are also not returning anything e.g using groupby with select

[#5]

@Sharmadha Sainath could you please also tell me how does this works

`Log Data` ??? what am I supposed to write in single quotes and how do you explain the results?

[#6]

@Saba Baig, you can search for any entities tagged with a particular tag name using the single quotes. for e.g. `Log Data` searches for all entities in atlas tagged as Log Data. Only classification name is supported in single quotes.

[#7]

@Sharmadha Sainath can you please guide me on this? as the method "searchUsingDslQuery" from EntityDiscoveryService.java is getting the result of the fetched attributes but then it only adds the attributes to the object of AtlasSearchResult...neither an AtlasVertex nor contains theInstance or theTrait.... Could you please guide me on this?

[#8]

@Saba BaigFew aggregate functions are not working in V2 APIs.

There are few Apache JIRAs filed for the same :

https://issues.apache.org/jira/browse/ATLAS-1874

https://issues.apache.org/jira/browse/ATLAS-1639

However V1 APIs work perfectly .

For example :

Query : hive_table groupby(owner) select owner,count()

URL encoded :

http://localhost:21000/api/atlas/discovery/search/dsl?limit=25&query=hive_table+groupby(owner)+selec... 
Response :

{requestId: "pool-2-thread-9 - 8a45b7a6-0cd9-4e9d-8b71-66e313f5abf1",
query: "hive_table groupby(owner) select owner,count()",
queryType: "dsl",
count: 2,
results: 
[
{$typeName$: "__tempQueryResultStruct74",
count(): 9,
owner: "hrt_qa"
},

{$typeName$: "__tempQueryResultStruct74",
count(): 2,
owner: "anonymous"
}
],
dataType: 
{typeName: "__tempQueryResultStruct74",
typeDescription: null,
typeVersion: "1.0",
attributeDefinitions: 
[
{name: "owner",
dataTypeName: "string",
multiplicity: 
{lower: 0,
upper: 1,
isUnique: false
},
isComposite: false,
isUnique: false,
isIndexable: false,
reverseAttributeName: null
},

{name: "count()",
dataTypeName: "long",
multiplicity: 
{lower: 0,
upper: 1,
isUnique: false
},
isComposite: false,
isUnique: false,
isIndexable: false,
reverseAttributeName: null
}
]
}
}

This query groups the owner of hive_table and and lists the owner ,count in each group . The concept is similar to that of SQL.

Since the UI currently uses V2 query , you may not see the results on UI. But you can always rely on REST APIs.

For this particular kind of queries , please use V1 APIs for now .

[#9]

@Saba Baig

>> Also, what features e.g wildcards (? . * etc) are available in full-text provided by Atlas? and how can we use them (examples)?

Atlas uses lucene query syntax for its full text search (basic search).

Please refer lucene query usage here :

http://lucene.apache.org/core/4_1_0/queryparser/org/apache/lucene/queryparser/classic/package-summar...

Example:

If I have 3 hive_table entities in Atlas , table_a , table_b , table_ab

Query :
1)
type = hive_table , query = table_?

http://localhost:21000/api/atlas/v2/search/basic?limit=25&excludeDeletedEntities=true&query=table_?&...

Returns 2 entities (table_a , table_b)

2)
type = hive_table , query = table_*

http://localhost:21000/api/atlas/v2/search/basic?limit=25&excludeDeletedEntities=true&query=table_*&...

Returns 3 entities (table_a , table_b , table_ab)

I hope this example makes it clear ...

Note : Wild card search can be done on UI itself.

[#10]

@Sharmadha Sainath

I have an entity of type "Table" named as "sales_fact". I tried to get that using api

http://localhost:21000/api/atlas/v2/search/basic?limit=25&includeDeletedEntities=true&query=sales_fa...

=====Response =====
{  "queryType": "BASIC",  "type": "Table",  "entities": [  {  "typeName": "Table",  "attributes": {  "owner": "Joe",  "qualifiedName": "sales_fact",  "name": "sales_fact",  "description": "sales fact table"  },  "guid": "c9838518-041f-4759-b3e5-01147b1526d1",  "status": "ACTIVE",  "displayText": "sales_fact",  "classificationNames": [  "Fact"  ]  }  ]}

But when I tried to the same search (Basic using query) I typed sales_fa?t and I got invalid Expression. I added a breakpoint in the API but it was never called. I think there is some validation applied on that field that omits any character like ? from the query before submitting it to API. Please guide.

I think, this could be the reason

Apache Lucene Analysis Pipeline

[#11]

@Saba Baig ,

>> But when I tried to the same search (Basic using query) I typed sales_fa?t and I got invalid Expression

What is the query you fired ?

You mentioned http://localhost:21000/api/atlas/v2/search/basic?limit=25∈cludeDeletedEntities=true&query=sales_fa?t... works for you. I am not able to get your question.

[#12]

@Sharmadha Sainath I said that I got invalid expression when fired the same query in the Atlas UI for basic search using query. Here it gave me Invalid Expression for the same query (sales_fa?t) that worked via api as mentioned earlier.

[#13]

@Saba Baig

Please refer to this issue

https://issues.apache.org/jira/browse/ATLAS-1788

[#14]

@Sharmadha Sainath

You also mentioned that if we write anything in single quotes it gives you all the entities with that tag. I found this behaving the same in case of DSL (Advanced Search)

Search for `Fact` in Basic Search

But, searching for `Fact` in Basic (Search by Query) behaved differently...

Question 1: If we follow what you just said then why am I seeing Log Data and Metric appearing there?

Question 2: Secondly, you said "Only classification name is supported in single quotes." then,

2.1) What is a classification name?

2.2) Difference between tag and classification name?

2.3) What to mention in single quotes... Classification name or tag?

[#15]

2.1) What is a classification name?

2.2) Difference between tag and classification name?

Classification and tags are same

Basic search is basically Lucene search . So ,If you see your result in the screenshot, basic search has returned everything related to fact.(i.e) entities with description containing fact.

[#16]

Thanks @Sharmadha Sainath