Reply
Explorer
Posts: 13
Registered: ‎09-05-2018

Using REST API, how to search an operation with originalName

I am trying to search an operation by its originalName("FROM census INSERT OVERWRITE LOCAL DI...2012(Stage") and saw veired result. For the below two queries i am seeing the results which are not related to this operation. Can someone help how to use the api to search for a particular operation based on originalName. 

 

http://10.33.6.155:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:...

http://10.33.6.155:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:...

 

 

Cloudera Employee
Posts: 41
Registered: ‎10-07-2016

Re: Using REST API, how to search an operation with originalName

Thanks for reaching out to the cloudera community. 

 

I noticed in your query, you've included an elipses. 

("FROM census INSERT OVERWRITE LOCAL DI...2012(Stage") 

 

Is this intended, or the result of truncation? if it's truncated, we'll need the full query that you're sending to the api. 

 

Also, can we get an example of the output you're getting? 

Explorer
Posts: 13
Registered: ‎09-05-2018

Re: Using REST API, how to search an operation with originalName

Elipses is not intensional. It is part of the originalName. Below is the response message for the links provided below.

 

http://10.33.6.146:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:...

http://10.33.6.146:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:...

 

[ {
  "identity" : "6d8026a6f2f0223a6b5d2de8c0921315",
  "originalName" : "INSERT OVERWRITE DIRECTORY '/user/cloudertest' SELEC... Part 3",
  "originalDescription" : null,
  "sourceId" : "9c19b614f83ef16154f7cf9490d68ada",
  "firstClassParentId" : "666931f137596053971458e42f1d03dc",
  "parentPath" : null,
  "extractorRunId" : "1b6d559ba1aeb8057fce2ea46a785d07##1707",
  "customProperties" : null,
  "name" : null,
  "description" : null,
  "tags" : null,
  "properties" : null,
  "technicalProperties" : null,
  "index" : null,
  "type" : "SUB_OPERATION",
  "userEntity" : false,
  "deleted" : null,
  "sourceType" : "HIVE",
  "packageName" : "nav",
  "metaClassName" : "hv_query_part",
  "internalType" : "hv_query_part"
}, {
  "identity" : "1df837140b1cfff6a3f4f2138b572e63",
  "originalName" : "INSERT OVERWRITE DIRECTORY '/user/cloudertest' SELEC... Part 4",
  "originalDescription" : null,
  "sourceId" : "9c19b614f83ef16154f7cf9490d68ada",
  "firstClassParentId" : "666931f137596053971458e42f1d03dc",
  "parentPath" : null,
  "extractorRunId" : "1b6d559ba1aeb8057fce2ea46a785d07##1707",
  "customProperties" : null,
  "name" : null,
  "description" : null,
  "tags" : null,
  "properties" : null,
  "technicalProperties" : null,
  "index" : null,
  "type" : "SUB_OPERATION",
  "userEntity" : false,
  "deleted" : null,
  "sourceType" : "HIVE",
  "packageName" : "nav",
  "metaClassName" : "hv_query_part",
  "internalType" : "hv_query_part"
} ]

 

Cloudera Employee
Posts: 3
Registered: ‎12-18-2017

Re: Using REST API, how to search an operation with originalName

Hi Nukala,

 

I used the "originalName" from the output you gave in the Swagger interactive API for Navigator on Cloudera Manager 5.9. Given a query string of:

 

sourceType:yarn and type:operation and originalName:"INSERT OVERWRITE DIRECTORY '/user/cloudertest' SELEC... Part 3"

where I've left out the extra parentheses and left in the spaces. It generates:

 

https://<host>:7187/api/v9/entities/?query=sourceType%3Ayarn%20and%20type%3Aoperation%20and%20originalName%3A%22INSERT%20OVERWRITE%20DIRECTORY%20'%2Fuser%2Fcloudertest'%20SELEC...%20Part%203%22&limit=100&offset=0
Here's your version:
http://10.33.6.146:7187/api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:FROM+census+INSERT+OVERWRITE+LOCAL+DI...2012%28Stage))&limit=10&offset=10
There's enough differences here that maybe it will give you some ideas for how you might change your URL to form the appropriate query.
 
I played around with it a bit and I was able to access identifiers with ellipses and with double quotes. As you probably found, the double quotes are stored in Navigator with a back slash escape that isn't displayed in the Navigator console.
I hope this helps,
Carol
Explorer
Posts: 13
Registered: ‎09-05-2018

Re: Using REST API, how to search an operation with originalName

Hi Carol,

Let's rephrase the question.

We have the following operation: FROM census INSERT OVERWRITE LOCAL DI...2012(Stage

image.png

It was created as a result of some query execution and now we want to see this exactly operation as a result of query to Cloudera Navigator API

Output should be like this:

[{
  "originalName": "FROM census INSERT OVERWRITE LOCAL DI...2012(Stage",
  "originalDescription": null,
  "sourceId": "8",
  "firstClassParentId": null,
  "parentPath": null,
  "deleteTime": null,
  "extractorRunId": "8##1828",
  "customProperties": null,
  "name": "FROM census INSERT OVERWRITE LOCAL DI...2012(Stage",
  "description": null,
  "tags": null,
  "properties": null,
  "technicalProperties": null,
  "wfIds": null,
  "inputFormat": "org.apache.hadoop.hive.ql.io.CombineHiveInputFormat",
  "outputFormat": "org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl",
  "outputKey": "org.apache.hadoop.io.Text",
  "outputValue": "org.apache.hadoop.io.Text",
  "mapper": "org.apache.hadoop.hive.ql.exec.mr.ExecMapper",
  "reducer": "org.apache.hadoop.hive.ql.exec.mr.ExecReducer",
  "sourceType": "YARN",
  "type": "OPERATION",
  "userEntity": false,
  "metaClassName": "mrjobspec",
  "deleted": null,
  "packageName": "nav",
  "identity": "123456",
  "internalType": "mrjobspec"
}]

In query URL we are specifying sourceType:YARN and type:operation.

What should we use as originalName in order to get output mentioned above?

If I just copy\paste the originalName from output it returns very strange irrelevant output:

api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:FROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage)):

image.png

This output contains directories and tables but the query URL explicitly specifies we want to see (sourceType:YARN)AND(type:operation).

Is it a bug?

If yes is it going to be fixed? Or some workaround exists?

Cloudera Employee
Posts: 3
Registered: ‎12-18-2017

Re: Using REST API, how to search an operation with originalName

Nukala,

I'm able to reproduce what you are seeing: the call is returning random stuff because it isn't parsing the query. I believe you can fix this by replacing the colons with "%3A".

 

That is, instead of this:

api/v9/entities?query=(sourceType:YARN)AND(type:operation)AND((originalName:FROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage))

 

try this:

api/v9/entities?query=(sourceType%3AYARN)AND(type%3Aoperation)AND((originalName%3AFROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage))

 

If I've missed your point, please make sure you get expected results with only the sourceType and type (you can limit the results to 3 or 4):

 

(sourceType:YARN)AND(type:operation)

vs

(sourceType%3AYARN)AND(type%3Aoperation)

Also, you might try just the originalName (skip the other entries for the moment).

Also, try a different entity type. That is, get the query working and then swap out the specifics for this entity.

Carol

Explorer
Posts: 13
Registered: ‎09-05-2018

Re: Using REST API, how to search an operation with originalName

Hello, 

api/v9/entities?query=(sourceType%3AYARN)AND(type%3Aoperation)AND((originalName%3AFROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage))

does not work as well: shows all those trash regardless sourceTypa and type

 

Cloudera Employee
Posts: 3
Registered: ‎12-18-2017

Re: Using REST API, how to search an operation with originalName

Did you get a chance to try some partial queries?

...try just the originalName (skip the other entries for the moment).

...try a different entity type.

That is, get the query working and then swap out the specifics for this entity.

Highlighted
Explorer
Posts: 13
Registered: ‎09-05-2018

Re: Using REST API, how to search an operation with originalName

Original name:

api/v9/entities?query=(originalName%3AFROM%20census%20INSERT%20OVERWRITE%20LOCAL%20DI...2012(Stage)

Does not work: shows trash as before.

Even if I specify different entity type.

Seems like some character(s) in that name just break everything.

Thiw works:

api/v9/entities?query=(originalName:FROM*census*INSERT*OVERWRITE*LOCAL*)

and this as well:

api/v9/entities?query=(originalName:FROM*census*INSERT*OVERWRITE*LOCAL*DI*2012*Stage)

Simply masking all the special chareacters with asterick seems to be helpful but this is dirty workaround

 

Announcements
New solutions