09-05-2018 11:20 PM
I am trying to search an operation by its originalName("FROM census INSERT OVERWRITE LOCAL DI...2012(Stage") and saw veired result. For the below two queries i am seeing the results which are not related to this operation. Can someone help how to use the api to search for a particular operation based on originalName.
09-18-2018 02:08 PM
Thanks for reaching out to the cloudera community.
I noticed in your query, you've included an elipses.
("FROM census INSERT OVERWRITE LOCAL DI...2012(Stage")
Is this intended, or the result of truncation? if it's truncated, we'll need the full query that you're sending to the api.
Also, can we get an example of the output you're getting?
09-19-2018 07:50 AM
Elipses is not intensional. It is part of the originalName. Below is the response message for the links provided below.
[ { "identity" : "6d8026a6f2f0223a6b5d2de8c0921315", "originalName" : "INSERT OVERWRITE DIRECTORY '/user/cloudertest' SELEC... Part 3", "originalDescription" : null, "sourceId" : "9c19b614f83ef16154f7cf9490d68ada", "firstClassParentId" : "666931f137596053971458e42f1d03dc", "parentPath" : null, "extractorRunId" : "1b6d559ba1aeb8057fce2ea46a785d07##1707", "customProperties" : null, "name" : null, "description" : null, "tags" : null, "properties" : null, "technicalProperties" : null, "index" : null, "type" : "SUB_OPERATION", "userEntity" : false, "deleted" : null, "sourceType" : "HIVE", "packageName" : "nav", "metaClassName" : "hv_query_part", "internalType" : "hv_query_part" }, { "identity" : "1df837140b1cfff6a3f4f2138b572e63", "originalName" : "INSERT OVERWRITE DIRECTORY '/user/cloudertest' SELEC... Part 4", "originalDescription" : null, "sourceId" : "9c19b614f83ef16154f7cf9490d68ada", "firstClassParentId" : "666931f137596053971458e42f1d03dc", "parentPath" : null, "extractorRunId" : "1b6d559ba1aeb8057fce2ea46a785d07##1707", "customProperties" : null, "name" : null, "description" : null, "tags" : null, "properties" : null, "technicalProperties" : null, "index" : null, "type" : "SUB_OPERATION", "userEntity" : false, "deleted" : null, "sourceType" : "HIVE", "packageName" : "nav", "metaClassName" : "hv_query_part", "internalType" : "hv_query_part" } ]
09-21-2018 06:13 PM
Hi Nukala,
I used the "originalName" from the output you gave in the Swagger interactive API for Navigator on Cloudera Manager 5.9. Given a query string of:
sourceType:yarn and type:operation and originalName:"INSERT OVERWRITE DIRECTORY '/user/cloudertest' SELEC... Part 3"
where I've left out the extra parentheses and left in the spaces. It generates:
https://<host>:7187/api/v9/entities/?query=sourceType%3Ayarn%20and%20type%3Aoperation%20and%20originalName%3A%22INSERT%20OVERWRITE%20DIRECTORY%20'%2Fuser%2Fcloudertest'%20SELEC...%20Part%203%22&limit=100&offset=0
http://10.33.6.146:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:FROM+census+INSERT+OVERWRITE+LOCAL+DI...2012%28Stage))&limit=10&offset=10
09-26-2018 02:52 AM
Hi Carol,
Let's rephrase the question.
We have the following operation: FROM census INSERT OVERWRITE LOCAL DI...2012(Stage
It was created as a result of some query execution and now we want to see this exactly operation as a result of query to Cloudera Navigator API
Output should be like this:
[{ "originalName": "FROM census INSERT OVERWRITE LOCAL DI...2012(Stage", "originalDescription": null, "sourceId": "8", "firstClassParentId": null, "parentPath": null, "deleteTime": null, "extractorRunId": "8##1828", "customProperties": null, "name": "FROM census INSERT OVERWRITE LOCAL DI...2012(Stage", "description": null, "tags": null, "properties": null, "technicalProperties": null, "wfIds": null, "inputFormat": "org.apache.hadoop.hive.ql.io.CombineHiveInputFormat", "outputFormat": "org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl", "outputKey": "org.apache.hadoop.io.Text", "outputValue": "org.apache.hadoop.io.Text", "mapper": "org.apache.hadoop.hive.ql.exec.mr.ExecMapper", "reducer": "org.apache.hadoop.hive.ql.exec.mr.ExecReducer", "sourceType": "YARN", "type": "OPERATION", "userEntity": false, "metaClassName": "mrjobspec", "deleted": null, "packageName": "nav", "identity": "123456", "internalType": "mrjobspec" }]
In query URL we are specifying sourceType:YARN and type:operation.
What should we use as originalName in order to get output mentioned above?
If I just copy\paste the originalName from output it returns very strange irrelevant output:
api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:FROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage)):
This output contains directories and tables but the query URL explicitly specifies we want to see (sourceType:YARN)AND(type:operation).
Is it a bug?
If yes is it going to be fixed? Or some workaround exists?
09-27-2018 11:50 AM
Nukala,
I'm able to reproduce what you are seeing: the call is returning random stuff because it isn't parsing the query. I believe you can fix this by replacing the colons with "%3A".
That is, instead of this:
api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:FROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage))
try this:
api/v9/entities?query=(sourceType%3AYARN)AND(type%3Aoperation)AND((originalName%3AFROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage))
If I've missed your point, please make sure you get expected results with only the sourceType and type (you can limit the results to 3 or 4):
(sourceType:YARN)AND(type:operation)
vs
(sourceType%3AYARN)AND(type%3Aoperation)
Also, you might try just the originalName (skip the other entries for the moment).
Also, try a different entity type. That is, get the query working and then swap out the specifics for this entity.
Carol
10-08-2018 02:14 AM
Hello,
api/v9/entities?query=(sourceType%3AYARN)AND(type%3Aoperation)AND((originalName%3AFROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage))
does not work as well: shows all those trash regardless sourceTypa and type
10-08-2018 08:49 AM
Did you get a chance to try some partial queries?
...try just the originalName (skip the other entries for the moment).
...try a different entity type.
That is, get the query working and then swap out the specifics for this entity.
10-11-2018 03:48 AM
Original name:
api/v9/entities?query=(originalName%3AFROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage)
Does not work: shows trash as before.
Even if I specify different entity type.
Seems like some character(s) in that name just break everything.
Thiw works:
api/v9/entities?query=(originalName:FROM*census*INSERT*OVERWRITE*LOCAL*)
and this as well:
api/v9/entities?query=(originalName:FROM*census*INSERT*OVERWRITE*LOCAL*DI*2012*Stage)
Simply masking all the special chareacters with asterick seems to be helpful but this is dirty workaround
10-19-2018 10:55 AM
Nukala, in reading through the information in the post and your latest reply, it looks as though you now have a workaround, but would like something a little cleaner if in fact it is possible to provide.
It seems the open questions are:
- What is the right way to mask the special characters?
- Is there a potential "global escape" character for the string so it could be used to begin and terminate the query string?
- Is there a better way to treat spaces in the query string?
Perry