Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is there a way of returning more than 100 results in DSL search API Atlas query?

Solved Go to solution
Highlighted

Is there a way of returning more than 100 results in DSL search API Atlas query?

Contributor

I can only return a maximum of 100 results when doing a DSL API search in Atlas. Is this by design or a bug?

Even with a limit of 1000 only 100 items are returned

curl -k -u admin:admin -H "Content-type:application/json" -X GET https://url:port/api/atlas/v2/search/dsl?query=hive_column%20where%20__state%3D%27ACTIVE%27%20and%20... | python -m json.tool > hive_column_prod_data_lake_limit.json

No limit yet still 100 items are returned. There are a lot more than 100 items - when I do the same for hive_tables only 100 items are returned.

curl -k -u admim:admin -H "Content-type:application/json" -X GET https://url:port/api/atlas/v2/earch/dsl?query=hive_column%20where%20__state%3D%27ACTIVE%27%20and%20q... | python -m json.tool > hive_column_prod_data_lake.json

This is on a 2.6.3 HDP install.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Is there a way of returning more than 100 results in DSL search API Atlas query?

Contributor

@Laura Ngo - by default the query returns maximum of 100 results. To retrieve more results, please add query parameter "limit" in the REST call, as shown below (addition of query parameter "limit=1234"):

curl -k -u admim:admin -H "Content-type:application/json"-X GET https://url:port/api/atlas/v2/earch/dsl?limit=1234&query=hive_column%20where%20__state%3D%27ACTIVE%2... | python -m json.tool > hive_column_prod_data_lake.json

Please note that a maximum of 10,000 results will be returned even if limit specified is higher.

Both default and the max limit can be configured with the following properties:

atlas.search.defaultlimit=100
atlas.search.maxlimit=10000

View solution in original post

2 REPLIES 2
Highlighted

Re: Is there a way of returning more than 100 results in DSL search API Atlas query?

Contributor

@Laura Ngo - by default the query returns maximum of 100 results. To retrieve more results, please add query parameter "limit" in the REST call, as shown below (addition of query parameter "limit=1234"):

curl -k -u admim:admin -H "Content-type:application/json"-X GET https://url:port/api/atlas/v2/earch/dsl?limit=1234&query=hive_column%20where%20__state%3D%27ACTIVE%2... | python -m json.tool > hive_column_prod_data_lake.json

Please note that a maximum of 10,000 results will be returned even if limit specified is higher.

Both default and the max limit can be configured with the following properties:

atlas.search.defaultlimit=100
atlas.search.maxlimit=10000

View solution in original post

Re: Is there a way of returning more than 100 results in DSL search API Atlas query?

Cloudera Employee

@Laura Ngo - the API shared by @Madhan Neethiraj is the correct one which you can use. However, in addition to it, I would like to inform you that if you have more than 10000 results and want to scroll between them you can use offset option like below query.

curl -k -u admim:admin -H "Content-type:application/json"-X GET https://url:port/api/atlas/v2/earch/dsl?limit=10000&offset=20000&query=hive_column%20where%20__state... | python -m json.tool > hive_column_prod_data_lake.json
Don't have an account?
Coming from Hortonworks? Activate your account here