Reply
Explorer
Posts: 19
Registered: ‎09-05-2018

Re: Cloudera Navigator Cluster Instance and Group

Thank you Elias.

 

Here is my understanding on Cluster, Cluster Group and Cluster Instances. Please correct me if I am wrong and also please clarify my doubts

 

1. Cluster Group and Cluster Instances are mostly related to AWS.

2. When you add  AWS Cluster as an external account to Cloudera, It will have a Cluster Group and Cluster Instance

3. Cluster Group will have multiple Cluster Instance which are same as different Clusters(AWS AMI's) added as external accounts to Cloudera from same AWS account

4. Suppose, I have a Cloudera Cluster then I have one Cluster Group and Cluster Instance.

- When I add AWS as an external account to the Cluster then I will have two Cluster Group and two Cluster Instances where the AWS Cluster Group will have the AWS Cluster Instance.

- Similarly if I add another AWS Cluster from the same account, then it will add another Cluster Instance to the same Cluster Group. So, now we will have two Cluster Groups and three Cluster Instances.

 

How we can identify different source types from different Clusters using Cloudera Navigator REST API's?

 

I see the following in Cloudera Navigator UI. Could you please how we can relate this with REST API response.

For Operation entities - in metadata it is having Cluster Group

Operation.JPG

 

 

http://10.33.6.155:7187/api/v9/entities?query=(sourceType:mapreduce)AND(originalName:wordcount)

 

[ {
  "originalName" : "wordcount",
  "originalDescription" : null,
  "sourceId" : "4692666",
  "firstClassParentId" : null,
  "parentPath" : null,
  "deleteTime" : null,
  "extractorRunId" : "4692666##525",
  "customProperties" : null,
  "name" : "wordcount",
  "description" : null,
  "tags" : null,
  "properties" : null,
  "technicalProperties" : null,
  "wfIds" : null,
  "inputFormat" : null,
  "outputFormat" : null,
  "outputKey" : "org.apache.hadoop.io.Text",
  "outputValue" : "org.apache.hadoop.io.IntWritable",
  "mapper" : "org.myorg.WordCount$Map",
  "reducer" : "org.myorg.WordCount$Reduce",
  "sourceType" : "MAPREDUCE",
  "type" : "OPERATION",
  "metaClassName" : "mrjobspec",
  "deleted" : null,
  "packageName" : "nav",
  "userEntity" : false,
  "identity" : "5431085",
  "internalType" : "mrjobspec"
} ]

 

For Operation Execution - in metadata it is having Cluster Instance

Operation_Execution.JPG

 

http://10.33.6.155:7187/api/v9/entities?query=(sourceType:mapreduce)AND(originalName:job_20181020202...)

 

[ {
  "originalName" : "job_201810202026_0003",
  "originalDescription" : null,
  "sourceId" : "4692666",
  "firstClassParentId" : null,
  "parentPath" : null,
  "deleteTime" : null,
  "extractorRunId" : "4692666##525",
  "customProperties" : null,
  "name" : null,
  "description" : null,
  "tags" : null,
  "properties" : null,
  "technicalProperties" : null,
  "started" : "2018-10-20T18:47:20.530Z",
  "ended" : "2018-10-20T18:47:50.778Z",
  "principal" : "cloudera",
  "inputs" : [ "hdfs://quickstart.cloudera:8020/user/cloudera/wordcount/input" ],
  "outputs" : [ "hdfs://quickstart.cloudera:8020/user/cloudera/wordcount/output13" ],
  "wfInstId" : null,
  "errorCodes" : null,
  "jobID" : "job_201810202026_0003",
  "sourceType" : "MAPREDUCE",
  "inputRecursive" : false,
  "type" : "OPERATION_EXECUTION",
  "metaClassName" : "mrjobinstance",
  "deleted" : null,
  "packageName" : "nav",
  "userEntity" : false,
  "identity" : "5435211",
  "internalType" : "mrjobinstance"
} ]

 

Cloudera Employee
Posts: 17
Registered: ‎08-22-2014

Re: Cloudera Navigator Cluster Instance and Group

Hi Nukala,

The following is from the docuentation I referenced in my last update:

 

"This new feature is supported with new properties and attributes displayed in the Cloudera Navigator console, to distinguish long-running from Altus clusters and to distinguish among transient clusters. For example, the Cluster Group property identifies all clusters that have been created using the same Altus Environment Name and Altus Cluster name. The Cluster Instance distinguishes each instance in the same group from the others, so that metadata and lineage extracted from transient clusters over time can be identified..."

 

For Atlus Clusters, please se our Cloudera Enterprise in the Cloud documentation

Announcements
New solutions