Cloudera Director 2.0 - Cluster Health Not Available

[#1]

Greetings,

I recently deployed Cloudera Director 2.0 and successfully pushed out a CDH 5 AWS cluster.

I am running into a wierd issue where Cloudera Director does not know the state of the cluster.

http://i.imgur.com/4fxmk1o.png 

While the cluster shows perfectly healthy in Cloudera Manager.

http://i.imgur.com/gBSb6Vc.png

I verified that Cloudera Director can SSH into the other instances without issue.

Any thoughts?

I see no errors in the Cloudera Director application logs other than an INFO level log line "RequestMappingInfo did not match any include patterns"

The logs: http://pastebin.com/35D7tspS  

[#2]

Hello,

Are you seeing the unknown health consistently, or is it intermittent? If it's consistent, is it possible the credentials were changed on Cloudera Manager? If the password changed, it's possible Director cannot login and fetch the required metrics.

If that's not the case, could you also check the status of the cluster via the API? The example URL would be:

http://<directorhost>:7189/api/v4/environments/<environment_name>/deployments/<demployment_name>/clusters/<cluster_name>/ This will list the specific status for every service. 

Thanks,

Jayita

[#3]

Greetings,

The issue is persistant. We did change the default admin password in Cloudera Manager (necessary to keep the environment secure, or anyone in the company could login). 

Is there a way I can make Director aware of a password change? It was also unable to see the status before the psw change. 

[#4]

I have run the API command you suggested that dumps the cluster status. This was the result.

[ec2-user@ip-10-170-0-201 ~]$ curl --user admin:<redacted> http://10.170.0.201:7189/api/v4/environments/New%20Platform/deployments/CM01/clusters/
[ "Cluster01" ][ec2-user@ip-10-170-0-201 ~]$ curl --user admin:CDH_g0liv3 http://10.170.0.201:7189/api/v4/environments/New%20Platform/deployments/CM01/clusters/Cluster01
{
"name" : "Cluster01",
"instances" : [ {
"ipAddress" : "10.170.12.140",
"properties" : {
"subnetId" : "subnet-1e1f6f47",
"ebsOptimized" : "false",
"virtualizationType" : "hvm",
"imageId" : "ami-414b7271",
"instanceType" : "m4.large",
"keyName" : "NewPlatformRoot",
"publicDnsName" : "",
"sourceDestCheck" : "true",
"availabilityZone" : "us-west-2c",
"privateIpAddress" : "10.170.12.140",
"rootDeviceType" : "ebs",
"launchTime" : "Fri Jan 22 09:59:22 EST 2016",
"instanceId" : "i-709e08aa",
"placementGroupName" : "",
"hypervisor" : "xen",
"spot" : "false",
"vpcId" : "vpc-1b47767e",
"rootDeviceName" : "/dev/sda1",
"privateDnsName" : "ip-10-170-12-140.us-west-2.compute.internal",
"securityGroups" : "launch-wizard-1(sg-fe2aca99)",
"architecture" : "x86_64"
},
"virtualInstance" : {
"id" : "cce0aec6-3eaf-4126-8ccc-63b1e4af4d9b",
"template" : {
"name" : "ClouderaManager",
"type" : "m4.large",
"image" : "ami-414b7271",
"bootstrapScript" : "",
"config" : {
"subnetId" : "subnet-1e1f6f47",
"rootVolumeType" : "gp2",
"instanceNamePrefix" : "director",
"rootVolumeSizeGB" : "50",
"useSpotInstances" : "false",
"securityGroupsIds" : "sg-fe2aca99"
},
"tags" : {
"Name" : "NewPlatform-ClouderaManager"
},
"normalizeInstance" : true
}
},
"capabilities" : {
"operatingSystemType" : "REDHAT_COMPATIBLE",
"operatingSystemVersion" : "REDHAT_COMPATIBLE_6",
"virtualizationType" : "HARDWARE_ASSISTED",
"pythonVersion" : "2.6.6",
"packageManager" : "YUM",
"passwordlessSudoEnabled" : true,
"selinuxEnabled" : true,
"iptablesEnabled" : false
},
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
}
}, {
"ipAddress" : "10.170.1.109",
"properties" : {
"subnetId" : "subnet-1e1f6f47",
"ebsOptimized" : "false",
"virtualizationType" : "hvm",
"imageId" : "ami-414b7271",
"instanceType" : "m4.large",
"keyName" : "NewPlatformRoot",
"publicDnsName" : "",
"sourceDestCheck" : "true",
"availabilityZone" : "us-west-2c",
"privateIpAddress" : "10.170.1.109",
"rootDeviceType" : "ebs",
"launchTime" : "Fri Jan 22 09:59:22 EST 2016",
"instanceId" : "i-7f9e08a5",
"placementGroupName" : "",
"hypervisor" : "xen",
"spot" : "false",
"vpcId" : "vpc-1b47767e",
"rootDeviceName" : "/dev/sda1",
"privateDnsName" : "ip-10-170-1-109.us-west-2.compute.internal",
"securityGroups" : "launch-wizard-1(sg-fe2aca99)",
"architecture" : "x86_64"
},
"virtualInstance" : {
"id" : "ccdc7ab5-f6ef-4f46-be65-1f4664b42433",
"template" : {
"name" : "ClouderaManager",
"type" : "m4.large",
"image" : "ami-414b7271",
"bootstrapScript" : "",
"config" : {
"subnetId" : "subnet-1e1f6f47",
"rootVolumeType" : "gp2",
"instanceNamePrefix" : "director",
"rootVolumeSizeGB" : "50",
"useSpotInstances" : "false",
"securityGroupsIds" : "sg-fe2aca99"
},
"tags" : {
"Name" : "NewPlatform-ClouderaManager"
},
"normalizeInstance" : true
}
},
"capabilities" : {
"operatingSystemType" : "REDHAT_COMPATIBLE",
"operatingSystemVersion" : "REDHAT_COMPATIBLE_6",
"virtualizationType" : "HARDWARE_ASSISTED",
"pythonVersion" : "2.6.6",
"packageManager" : "YUM",
"passwordlessSudoEnabled" : true,
"selinuxEnabled" : true,
"iptablesEnabled" : false
},
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
}
}, {
"ipAddress" : "10.170.1.108",
"properties" : {
"subnetId" : "subnet-1e1f6f47",
"ebsOptimized" : "false",
"virtualizationType" : "hvm",
"imageId" : "ami-414b7271",
"instanceType" : "m4.large",
"keyName" : "NewPlatformRoot",
"publicDnsName" : "",
"sourceDestCheck" : "true",
"availabilityZone" : "us-west-2c",
"privateIpAddress" : "10.170.1.108",
"rootDeviceType" : "ebs",
"launchTime" : "Fri Jan 22 09:59:22 EST 2016",
"instanceId" : "i-7c9e08a6",
"placementGroupName" : "",
"hypervisor" : "xen",
"spot" : "false",
"vpcId" : "vpc-1b47767e",
"rootDeviceName" : "/dev/sda1",
"privateDnsName" : "ip-10-170-1-108.us-west-2.compute.internal",
"securityGroups" : "launch-wizard-1(sg-fe2aca99)",
"architecture" : "x86_64"
},
"virtualInstance" : {
"id" : "2ff16e18-24e7-4352-92f8-b74e292c4d90",
"template" : {
"name" : "ClouderaManager",
"type" : "m4.large",
"image" : "ami-414b7271",
"bootstrapScript" : "",
"config" : {
"subnetId" : "subnet-1e1f6f47",
"rootVolumeType" : "gp2",
"instanceNamePrefix" : "director",
"rootVolumeSizeGB" : "50",
"useSpotInstances" : "false",
"securityGroupsIds" : "sg-fe2aca99"
},
"tags" : {
"Name" : "NewPlatform-ClouderaManager"
},
"normalizeInstance" : true
}
},
"capabilities" : {
"operatingSystemType" : "REDHAT_COMPATIBLE",
"operatingSystemVersion" : "REDHAT_COMPATIBLE_6",
"virtualizationType" : "HARDWARE_ASSISTED",
"pythonVersion" : "2.6.6",
"packageManager" : "YUM",
"passwordlessSudoEnabled" : true,
"selinuxEnabled" : true,
"iptablesEnabled" : false
},
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
}
}, {
"ipAddress" : "10.170.1.110",
"properties" : {
"subnetId" : "subnet-1e1f6f47",
"ebsOptimized" : "false",
"virtualizationType" : "hvm",
"imageId" : "ami-414b7271",
"instanceType" : "m4.large",
"keyName" : "NewPlatformRoot",
"publicDnsName" : "",
"sourceDestCheck" : "true",
"availabilityZone" : "us-west-2c",
"privateIpAddress" : "10.170.1.110",
"rootDeviceType" : "ebs",
"launchTime" : "Fri Jan 22 09:59:22 EST 2016",
"instanceId" : "i-7d9e08a7",
"placementGroupName" : "",
"hypervisor" : "xen",
"spot" : "false",
"vpcId" : "vpc-1b47767e",
"rootDeviceName" : "/dev/sda1",
"privateDnsName" : "ip-10-170-1-110.us-west-2.compute.internal",
"securityGroups" : "launch-wizard-1(sg-fe2aca99)",
"architecture" : "x86_64"
},
"virtualInstance" : {
"id" : "9bdde564-c490-45ad-864b-172f3907ee0a",
"template" : {
"name" : "ClouderaManager",
"type" : "m4.large",
"image" : "ami-414b7271",
"bootstrapScript" : "",
"config" : {
"subnetId" : "subnet-1e1f6f47",
"rootVolumeType" : "gp2",
"instanceNamePrefix" : "director",
"rootVolumeSizeGB" : "50",
"useSpotInstances" : "false",
"securityGroupsIds" : "sg-fe2aca99"
},
"tags" : {
"Name" : "NewPlatform-ClouderaManager"
},
"normalizeInstance" : true
}
},
"capabilities" : {
"operatingSystemType" : "REDHAT_COMPATIBLE",
"operatingSystemVersion" : "REDHAT_COMPATIBLE_6",
"virtualizationType" : "HARDWARE_ASSISTED",
"pythonVersion" : "2.6.6",
"packageManager" : "YUM",
"passwordlessSudoEnabled" : true,
"selinuxEnabled" : true,
"iptablesEnabled" : false
},
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
}
}, {
"ipAddress" : "10.170.1.107",
"properties" : {
"subnetId" : "subnet-1e1f6f47",
"ebsOptimized" : "false",
"virtualizationType" : "hvm",
"imageId" : "ami-414b7271",
"instanceType" : "m4.large",
"keyName" : "NewPlatformRoot",
"publicDnsName" : "",
"sourceDestCheck" : "true",
"availabilityZone" : "us-west-2c",
"privateIpAddress" : "10.170.1.107",
"rootDeviceType" : "ebs",
"launchTime" : "Fri Jan 22 09:59:22 EST 2016",
"instanceId" : "i-729e08a8",
"placementGroupName" : "",
"hypervisor" : "xen",
"spot" : "false",
"vpcId" : "vpc-1b47767e",
"rootDeviceName" : "/dev/sda1",
"privateDnsName" : "ip-10-170-1-107.us-west-2.compute.internal",
"securityGroups" : "launch-wizard-1(sg-fe2aca99)",
"architecture" : "x86_64"
},
"virtualInstance" : {
"id" : "b3ca74d7-4a82-49d2-8dbd-44cac2fad24a",
"template" : {
"name" : "ClouderaManager",
"type" : "m4.large",
"image" : "ami-414b7271",
"bootstrapScript" : "",
"config" : {
"subnetId" : "subnet-1e1f6f47",
"rootVolumeType" : "gp2",
"instanceNamePrefix" : "director",
"rootVolumeSizeGB" : "50",
"useSpotInstances" : "false",
"securityGroupsIds" : "sg-fe2aca99"
},
"tags" : {
"Name" : "NewPlatform-ClouderaManager"
},
"normalizeInstance" : true
}
},
"capabilities" : {
"operatingSystemType" : "REDHAT_COMPATIBLE",
"operatingSystemVersion" : "REDHAT_COMPATIBLE_6",
"virtualizationType" : "HARDWARE_ASSISTED",
"pythonVersion" : "2.6.6",
"packageManager" : "YUM",
"passwordlessSudoEnabled" : true,
"selinuxEnabled" : true,
"iptablesEnabled" : false
},
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
}
}, {
"ipAddress" : "10.170.1.106",
"properties" : {
"subnetId" : "subnet-1e1f6f47",
"ebsOptimized" : "false",
"virtualizationType" : "hvm",
"imageId" : "ami-414b7271",
"instanceType" : "m4.large",
"keyName" : "NewPlatformRoot",
"publicDnsName" : "",
"sourceDestCheck" : "true",
"availabilityZone" : "us-west-2c",
"privateIpAddress" : "10.170.1.106",
"rootDeviceType" : "ebs",
"launchTime" : "Fri Jan 22 09:59:22 EST 2016",
"instanceId" : "i-739e08a9",
"placementGroupName" : "",
"hypervisor" : "xen",
"spot" : "false",
"vpcId" : "vpc-1b47767e",
"rootDeviceName" : "/dev/sda1",
"privateDnsName" : "ip-10-170-1-106.us-west-2.compute.internal",
"securityGroups" : "launch-wizard-1(sg-fe2aca99)",
"architecture" : "x86_64"
},
"virtualInstance" : {
"id" : "d097a064-82a1-4ef8-b108-fd64edb0450d",
"template" : {
"name" : "ClouderaManager",
"type" : "m4.large",
"image" : "ami-414b7271",
"bootstrapScript" : "",
"config" : {
"subnetId" : "subnet-1e1f6f47",
"rootVolumeType" : "gp2",
"instanceNamePrefix" : "director",
"rootVolumeSizeGB" : "50",
"useSpotInstances" : "false",
"securityGroupsIds" : "sg-fe2aca99"
},
"tags" : {
"Name" : "NewPlatform-ClouderaManager"
},
"normalizeInstance" : true
}
},
"capabilities" : {
"operatingSystemType" : "REDHAT_COMPATIBLE",
"operatingSystemVersion" : "REDHAT_COMPATIBLE_6",
"virtualizationType" : "HARDWARE_ASSISTED",
"pythonVersion" : "2.6.6",
"packageManager" : "YUM",
"passwordlessSudoEnabled" : true,
"selinuxEnabled" : true,
"iptablesEnabled" : false
},
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
}
}, {
"ipAddress" : "10.170.0.142",
"properties" : {
"subnetId" : "subnet-1e1f6f47",
"ebsOptimized" : "false",
"virtualizationType" : "hvm",
"imageId" : "ami-414b7271",
"instanceType" : "m4.large",
"keyName" : "NewPlatformRoot",
"publicDnsName" : "",
"sourceDestCheck" : "true",
"availabilityZone" : "us-west-2c",
"privateIpAddress" : "10.170.0.142",
"rootDeviceType" : "ebs",
"launchTime" : "Fri Jan 22 09:59:22 EST 2016",
"instanceId" : "i-f8910722",
"placementGroupName" : "",
"hypervisor" : "xen",
"spot" : "false",
"vpcId" : "vpc-1b47767e",
"rootDeviceName" : "/dev/sda1",
"privateDnsName" : "ip-10-170-0-142.us-west-2.compute.internal",
"securityGroups" : "launch-wizard-1(sg-fe2aca99)",
"architecture" : "x86_64"
},
"virtualInstance" : {
"id" : "acaa5119-b13f-4051-8841-7567e15e30e5",
"template" : {
"name" : "ClouderaManager",
"type" : "m4.large",
"image" : "ami-414b7271",
"bootstrapScript" : "",
"config" : {
"subnetId" : "subnet-1e1f6f47",
"rootVolumeType" : "gp2",
"instanceNamePrefix" : "director",
"rootVolumeSizeGB" : "50",
"useSpotInstances" : "false",
"securityGroupsIds" : "sg-fe2aca99"
},
"tags" : {
"Name" : "NewPlatform-ClouderaManager"
},
"normalizeInstance" : true
}
},
"capabilities" : {
"operatingSystemType" : "REDHAT_COMPATIBLE",
"operatingSystemVersion" : "REDHAT_COMPATIBLE_6",
"virtualizationType" : "HARDWARE_ASSISTED",
"pythonVersion" : "2.6.6",
"packageManager" : "YUM",
"passwordlessSudoEnabled" : true,
"selinuxEnabled" : true,
"iptablesEnabled" : false
},
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
}
} ],
"services" : [ {
"serviceName" : "ZOOKEEPER",
"url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-ZOOKEEPER-OwinYWHV",
"roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-ZOOKEEPER-OwinYWHV/ins...
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
},
"healthChecks" : [ ]
}, {
"serviceName" : "SQOOP",
"url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SQOOP-UrPslgqE",
"roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SQOOP-UrPslgqE/instanc...
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
},
"healthChecks" : [ ]
}, {
"serviceName" : "SPARK_ON_YARN",
"url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SPARK_ON_YARN-AJfBtbso...
"roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SPARK_ON_YARN-AJfBtbso...
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
},
"healthChecks" : [ ]
}, {
"serviceName" : "HIVE",
"url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HIVE-QAFjCOQn",
"roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HIVE-QAFjCOQn/instance...
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
},
"healthChecks" : [ ]
}, {
"serviceName" : "OOZIE",
"url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-OOZIE-wMmgwoTc",
"roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-OOZIE-wMmgwoTc/instanc...
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
},
"healthChecks" : [ ]
}, {
"serviceName" : "HUE",
"url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HUE-UBSUTqcp",
"roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HUE-UBSUTqcp/instances...
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
},
"healthChecks" : [ ]
}, {
"serviceName" : "YARN",
"url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-YARN-Cahmhgyy",
"roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-YARN-Cahmhgyy/instance...
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
},
"healthChecks" : [ ]
}, {
"serviceName" : "HDFS",
"url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HDFS-Fxvxaehf",
"roleInstancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HDFS-Fxvxaehf/instance...
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453733448326
},
"healthChecks" : [ ]
} ],
"url" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/clusterRedirect/Cluster01",
"instancesUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/clusterRedirect/Cluster01/hosts",
"health" : {
"status" : "NOT_AVAILABLE",
"lastReported" : 1453821883390
}
}

[#5]

Here is the same information going directly through the Cloudera Manager API (instead of director). 

[ec2-user@ip-10-170-0-201 ~]$ curl --user admin:<redacted> http://10.170.2.146:7180/api/v7/clusters/Cluster01/services
{
"items" : [ {
"name" : "CD-HIVE-QAFjCOQn",
"type" : "HIVE",
"clusterRef" : {
"clusterName" : "Cluster01"
},
"serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HIVE-QAFjCOQn",
"serviceState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "HIVE_HIVEMETASTORES_HEALTHY",
"summary" : "GOOD"
}, {
"name" : "HIVE_HIVESERVER2S_HEALTHY",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "FRESH",
"clientConfigStalenessStatus" : "FRESH",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"displayName" : "HIVE-1"
}, {
"name" : "CD-HUE-UBSUTqcp",
"type" : "HUE",
"clusterRef" : {
"clusterName" : "Cluster01"
},
"serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HUE-UBSUTqcp",
"serviceState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "HUE_HUE_SERVERS_HEALTHY",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "FRESH",
"clientConfigStalenessStatus" : "FRESH",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"displayName" : "HUE-1"
}, {
"name" : "CD-OOZIE-wMmgwoTc",
"type" : "OOZIE",
"clusterRef" : {
"clusterName" : "Cluster01"
},
"serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-OOZIE-wMmgwoTc",
"serviceState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "OOZIE_OOZIE_SERVERS_HEALTHY",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "FRESH",
"clientConfigStalenessStatus" : "FRESH",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"displayName" : "OOZIE-1"
}, {
"name" : "CD-SPARK_ON_YARN-AJfBtbso",
"type" : "SPARK_ON_YARN",
"clusterRef" : {
"clusterName" : "Cluster01"
},
"serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SPARK_ON_YARN-AJfBtbso...
"serviceState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ ],
"configStalenessStatus" : "FRESH",
"clientConfigStalenessStatus" : "FRESH",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"displayName" : "SPARK_ON_YARN-1"
}, {
"name" : "CD-SQOOP-UrPslgqE",
"type" : "SQOOP",
"clusterRef" : {
"clusterName" : "Cluster01"
},
"serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-SQOOP-UrPslgqE",
"serviceState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "SQOOP_SQOOP_SERVER_HEALTH",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "FRESH",
"clientConfigStalenessStatus" : "FRESH",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"displayName" : "SQOOP-1"
}, {
"name" : "CD-YARN-Cahmhgyy",
"type" : "YARN",
"clusterRef" : {
"clusterName" : "Cluster01"
},
"serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-YARN-Cahmhgyy",
"serviceState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "YARN_JOBHISTORY_HEALTH",
"summary" : "GOOD"
}, {
"name" : "YARN_NODE_MANAGERS_HEALTHY",
"summary" : "GOOD"
}, {
"name" : "YARN_RESOURCEMANAGERS_HEALTH",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "FRESH",
"clientConfigStalenessStatus" : "FRESH",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"displayName" : "YARN-1"
}, {
"name" : "CD-HDFS-Fxvxaehf",
"type" : "HDFS",
"clusterRef" : {
"clusterName" : "Cluster01"
},
"serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-HDFS-Fxvxaehf",
"serviceState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "HDFS_BLOCKS_WITH_CORRUPT_REPLICAS",
"summary" : "GOOD"
}, {
"name" : "HDFS_CANARY_HEALTH",
"summary" : "GOOD"
}, {
"name" : "HDFS_DATA_NODES_HEALTHY",
"summary" : "GOOD"
}, {
"name" : "HDFS_FREE_SPACE_REMAINING",
"summary" : "GOOD"
}, {
"name" : "HDFS_HA_NAMENODE_HEALTH",
"summary" : "GOOD"
}, {
"name" : "HDFS_MISSING_BLOCKS",
"summary" : "GOOD"
}, {
"name" : "HDFS_UNDER_REPLICATED_BLOCKS",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "FRESH",
"clientConfigStalenessStatus" : "FRESH",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"displayName" : "HDFS-1"
}, {
"name" : "CD-ZOOKEEPER-OwinYWHV",
"type" : "ZOOKEEPER",
"clusterRef" : {
"clusterName" : "Cluster01"
},
"serviceUrl" : "http://ip-10-170-2-146.us-west-2.compute.internal:7180/cmf/serviceRedirect/CD-ZOOKEEPER-OwinYWHV",
"serviceState" : "STARTED",
"healthSummary" : "GOOD",
"healthChecks" : [ {
"name" : "ZOOKEEPER_CANARY_HEALTH",
"summary" : "GOOD"
}, {
"name" : "ZOOKEEPER_SERVERS_HEALTHY",
"summary" : "GOOD"
} ],
"configStalenessStatus" : "FRESH",
"clientConfigStalenessStatus" : "FRESH",
"maintenanceMode" : false,
"maintenanceOwners" : [ ],
"displayName" : "ZOOKEEPER-1"
} ]
}[ec2-user@ip-10-170-0-201 ~]$

[#6 · Accepted solution]

Hi,

Thanks for the information you provided. Since you have changed the Cloudera Manager credentials, it is possible to update Director to use the same, which should restore access, and have the health show up again.

On your Director, navigate to the status page for the deployment, i.e. for the CM. On the right there is a drop down menu, click on "Update Cloudera Manager Credentials". This should pop up a form where you can save the new credentials for Director. After a bit the health for the cluster will show up...

Hope this helps,

Jayita

[#7]

This worked! Thank you !!