question Re: Is there a way of returning more than 100 results in DSL search API Atlas query? in Archives of Support Questions (Read Only)

Is there a way of returning more than 100 results in DSL search API Atlas query?

laura_ngo — Sun, 11 Mar 2018 06:15:39 GMT

I can only return a maximum of 100 results when doing a DSL API search in Atlas. Is this by design or a bug?

Even with a limit of 1000 only 100 items are returned

curl -k -u admin:admin -H "Content-type:application/json" -X GET https://url:port/api/atlas/v2/search/dsl?query=hive_column%20where%20__state%3D%27ACTIVE%27%20and%20qualifiedName%20like%20%27prod_%2A_data_lake%2A%27%20select%20qualifiedName%2Cname%2C__guid%20limit%201000 | python -m json.tool > hive_column_prod_data_lake_limit.json

No limit yet still 100 items are returned. There are a lot more than 100 items - when I do the same for hive_tables only 100 items are returned.

curl -k -u admim:admin -H "Content-type:application/json" -X GET https://url:port/api/atlas/v2/earch/dsl?query=hive_column%20where%20__state%3D%27ACTIVE%27%20and%20qualifiedName%20like%20%27prod_%2A_data_lake%2A%27%20selct%20qualifiedName%2Cname%2C__guid | python -m json.tool > hive_column_prod_data_lake.json

This is on a 2.6.3 HDP install.

Re: Is there a way of returning more than 100 results in DSL search API Atlas query?

mneethiraj — Thu, 15 Mar 2018 22:55:21 GMT

@Laura Ngo - by default the query returns maximum of 100 results. To retrieve more results, please add query parameter "limit" in the REST call, as shown below (addition of query parameter "limit=1234"):

curl -k -u admim:admin -H "Content-type:application/json"-X GET https://url:port/api/atlas/v2/earch/dsl?limit=1234&query=hive_column%20where%20__state%3D%27ACTIVE%27%20and%20qualifiedName%20like%20%27prod_%2A_data_lake%2A%27%20selct%20qualifiedName%2Cname%2C__guid | python -m json.tool > hive_column_prod_data_lake.json

Please note that a maximum of 10,000 results will be returned even if limit specified is higher.

Both default and the max limit can be configured with the following properties:

atlas.search.defaultlimit=100
atlas.search.maxlimit=10000

Re: Is there a way of returning more than 100 results in DSL search API Atlas query?

rbansal — Fri, 16 Mar 2018 10:38:08 GMT

@Laura Ngo - the API shared by @Madhan Neethiraj is the correct one which you can use. However, in addition to it, I would like to inform you that if you have more than 10000 results and want to scroll between them you can use offset option like below query.

curl -k -u admim:admin -H "Content-type:application/json"-X GET https://url:port/api/atlas/v2/earch/dsl?limit=10000&offset=20000&query=hive_column%20where%20__state%3D%27ACTIVE%27%20and%20qualifiedName%20like%20%27prod_%2A_data_lake%2A%27%20selct%20qualifiedName%2Cname%2C__guid | python -m json.tool > hive_column_prod_data_lake.json