Using REST API, how to search an operation with originalName

[Nukala]

I am trying to search an operation by its originalName("FROM census INSERT OVERWRITE LOCAL DI...2012(Stage") and saw veired result. For the below two queries i am seeing the results which are not related to this operation. Can someone help how to use the api to search for a particular operation based on originalName. 

http://10.33.6.155:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:...

http://10.33.6.155:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:...

[jmartin1938]

Thanks for reaching out to the cloudera community. 

I noticed in your query, you've included an elipses. 

("FROM census INSERT OVERWRITE LOCAL DI...2012(Stage") 

Is this intended, or the result of truncation? if it's truncated, we'll need the full query that you're sending to the api. 

Also, can we get an example of the output you're getting? 

[Nukala]

Elipses is not intensional. It is part of the originalName. Below is the response message for the links provided below.

http://10.33.6.146:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:...

http://10.33.6.146:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:...

[ {
  "identity" : "6d8026a6f2f0223a6b5d2de8c0921315",
  "originalName" : "INSERT OVERWRITE DIRECTORY '/user/cloudertest' SELEC... Part 3",
  "originalDescription" : null,
  "sourceId" : "9c19b614f83ef16154f7cf9490d68ada",
  "firstClassParentId" : "666931f137596053971458e42f1d03dc",
  "parentPath" : null,
  "extractorRunId" : "1b6d559ba1aeb8057fce2ea46a785d07##1707",
  "customProperties" : null,
  "name" : null,
  "description" : null,
  "tags" : null,
  "properties" : null,
  "technicalProperties" : null,
  "index" : null,
  "type" : "SUB_OPERATION",
  "userEntity" : false,
  "deleted" : null,
  "sourceType" : "HIVE",
  "packageName" : "nav",
  "metaClassName" : "hv_query_part",
  "internalType" : "hv_query_part"
}, {
  "identity" : "1df837140b1cfff6a3f4f2138b572e63",
  "originalName" : "INSERT OVERWRITE DIRECTORY '/user/cloudertest' SELEC... Part 4",
  "originalDescription" : null,
  "sourceId" : "9c19b614f83ef16154f7cf9490d68ada",
  "firstClassParentId" : "666931f137596053971458e42f1d03dc",
  "parentPath" : null,
  "extractorRunId" : "1b6d559ba1aeb8057fce2ea46a785d07##1707",
  "customProperties" : null,
  "name" : null,
  "description" : null,
  "tags" : null,
  "properties" : null,
  "technicalProperties" : null,
  "index" : null,
  "type" : "SUB_OPERATION",
  "userEntity" : false,
  "deleted" : null,
  "sourceType" : "HIVE",
  "packageName" : "nav",
  "metaClassName" : "hv_query_part",
  "internalType" : "hv_query_part"
} ]

[cdrummond]

Hi Nukala,

I used the "originalName" from the output you gave in the Swagger interactive API for Navigator on Cloudera Manager 5.9. Given a query string of:

sourceType:yarn and type:operation and originalName:"INSERT OVERWRITE DIRECTORY '/user/cloudertest' SELEC... Part 3"

where I've left out the extra parentheses and left in the spaces. It generates:

https://<host>:7187/api/v9/entities/?query=sourceType%3Ayarn%20and%20type%3Aoperation%20and%20originalName%3A%22INSERT%20OVERWRITE%20DIRECTORY%20'%2Fuser%2Fcloudertest'%20SELEC...%20Part%203%22&limit=100&offset=0

Here's your version:

http://10.33.6.146:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:FROM+census+INSERT+OVERWRITE+LOCAL+DI...2012%28Stage))&limit=10&offset=10

There's enough differences here that maybe it will give you some ideas for how you might change your URL to form the appropriate query.

I played around with it a bit and I was able to access identifiers with ellipses and with double quotes. As you probably found, the double quotes are stored in Navigator with a back slash escape that isn't displayed in the Navigator console.

I hope this helps,

Carol

[Nukala]

Hi Carol,

Let's rephrase the question.

We have the following operation: FROM census INSERT OVERWRITE LOCAL DI...2012(Stage

It was created as a result of some query execution and now we want to see this exactly operation as a result of query to Cloudera Navigator API

Output should be like this:

[{
  "originalName": "FROM census INSERT OVERWRITE LOCAL DI...2012(Stage",
  "originalDescription": null,
  "sourceId": "8",
  "firstClassParentId": null,
  "parentPath": null,
  "deleteTime": null,
  "extractorRunId": "8##1828",
  "customProperties": null,
  "name": "FROM census INSERT OVERWRITE LOCAL DI...2012(Stage",
  "description": null,
  "tags": null,
  "properties": null,
  "technicalProperties": null,
  "wfIds": null,
  "inputFormat": "org.apache.hadoop.hive.ql.io.CombineHiveInputFormat",
  "outputFormat": "org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl",
  "outputKey": "org.apache.hadoop.io.Text",
  "outputValue": "org.apache.hadoop.io.Text",
  "mapper": "org.apache.hadoop.hive.ql.exec.mr.ExecMapper",
  "reducer": "org.apache.hadoop.hive.ql.exec.mr.ExecReducer",
  "sourceType": "YARN",
  "type": "OPERATION",
  "userEntity": false,
  "metaClassName": "mrjobspec",
  "deleted": null,
  "packageName": "nav",
  "identity": "123456",
  "internalType": "mrjobspec"
}]

In query URL we are specifying sourceType:YARN and type:operation.

What should we use as originalName in order to get output mentioned above?

If I just copy\paste the originalName from output it returns very strange irrelevant output:

api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:FROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage)):

This output contains directories and tables but the query URL explicitly specifies we want to see (sourceType:YARN)AND(type:operation).

Is it a bug?

If yes is it going to be fixed? Or some workaround exists?

[cdrummond]

Nukala,

I'm able to reproduce what you are seeing: the call is returning random stuff because it isn't parsing the query. I believe you can fix this by replacing the colons with "%3A".

That is, instead of this:

api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:FROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage))

try this:

api/v9/entities?query=(sourceType%3AYARN)AND(type%3Aoperation)AND((originalName%3AFROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage))

If I've missed your point, please make sure you get expected results with only the sourceType and type (you can limit the results to 3 or 4):

(sourceType:YARN)AND(type:operation)

vs

(sourceType%3AYARN)AND(type%3Aoperation)

Also, you might try just the originalName (skip the other entries for the moment).

Also, try a different entity type. That is, get the query working and then swap out the specifics for this entity.

Carol

[Nukala]

Hello, 

api/v9/entities?query=(sourceType%3AYARN)AND(type%3Aoperation)AND((originalName%3AFROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage))

does not work as well: shows all those trash regardless sourceTypa and type

[cdrummond]

Did you get a chance to try some partial queries?

...try just the originalName (skip the other entries for the moment).

...try a different entity type.

That is, get the query working and then swap out the specifics for this entity.

[Nukala]

Original name:

api/v9/entities?query=(originalName%3AFROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage)

Does not work: shows trash as before.

Even if I specify different entity type.

Seems like some character(s) in that name just break everything.

Thiw works:

api/v9/entities?query=(originalName:FROM*census*INSERT*OVERWRITE*LOCAL*)

and this as well:

api/v9/entities?query=(originalName:FROM*census*INSERT*OVERWRITE*LOCAL*DI*2012*Stage)

Simply masking all the special chareacters with asterick seems to be helpful but this is dirty workaround

[PerryOrne]

Nukala, in reading through the information in the post and your latest reply, it looks as though you now have a workaround, but would like something a little cleaner if in fact it is possible to provide.  

It seems the open questions are:

-  What is the right way to mask the special characters?

-  Is there a potential "global escape" character for the string so it could be used to begin and terminate the query string?

-  Is there a better way to treat spaces in the query string?

Perry