Cannot ping CM DeploymentKey

[#1]

Hi 

i setup the director on GCP and then added the environment using webui and created the deployment which resulted the error at the end cann't ping the CM deployment key. if i continue and add cluster all the nodes are provised but this error keeps on comming back. 

I am able to ssh from my local network to GCP and even able too ssh from director node to cloudera maanger

node using the ssh key provided while creating the environment. i am badly stuck. Here are lat line from the log.

2019-02-22 12:39:35.546 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.launchpad.cm.ApiCommandMonitor: Waiting for commands: [14]
[2019-02-22 12:40:16.167 +0000] INFO  [task-thread-7] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-22 12:40:16.168 +0000] INFO  [task-thread-7] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-22 12:40:16.190 +0000] INFO  [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2019-02-22 12:40:16.191 +0000] INFO  [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Skipping refresh of deployment env1:dep1 as it is in transition or terminated. (Stage: BOOTSTRAPPING, deploymentIsNull: true)
[2019-02-22 12:40:16.191 +0000] INFO  [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models
[2019-02-22 12:41:16.049 +0000] INFO  [task-thread-2] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-22 12:41:16.055 +0000] INFO  [task-thread-2] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-22 12:41:21.857 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.b.c.BootstrapClouderaManagerAgent: Command GlobalHostInstall with ID 14 completed successfully on instance 10.142.15.205. Details: ApiCommand{id=14, name=GlobalHostInstall, startTime=Fri Feb 22 12:39:34 UTC 2019, endTime=Fri Feb 22 12:41:12 UTC 2019, active=false, success=true, resultMessage=Successfully completed installation., serviceRef=null, roleRef=null, hostRef=null, parent=null}
[2019-02-22 12:41:27.841 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Setting up Cloudera Management Services
[2019-02-22 12:41:28.502 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.SetupClouderaManagementServices$CreateMgmtService - c.c.l.b.d.SetupClouderaManagementServices: Auto assigning management service roles
[2019-02-22 12:41:37.621 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.ConfigureClouderaManagementServices - c.c.l.b.d.ConfigureClouderaManagementServices: Restarting Cloudera Management Services: ApiCommand{id=24, name=Restart, startTime=Fri Feb 22 12:41:34 UTC 2019, endTime=null, active=true, success=null, resultMessage=null, serviceRef=ApiServiceRef{peerName=null, clusterName=null, serviceName=MGMT}, roleRef=null, hostRef=null, parent=null}
[2019-02-22 12:41:37.633 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Backing up Cloudera Manager Server configuration
[2019-02-22 12:41:37.767 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-22T12:56:37.767Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.15.205, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.15.205', hostAddress=Optional.of(/10.142.15.205)}, HostEndpoint{hostAddressString='34.73.239.227', hostAddress=Optional.of(/34.73.239.227)}]} Instance{virtualInstance=VirtualInstance{id='56daae7a-fca2-4b2c-ae15-f08806ffdd1e', template=InstanceTemplate{name='centos-6-9', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=10, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(4b1965e7-f35d-4680-898a-8588412fd98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-22T12:28:44.830Z, lastChecked=2019-02-22T12:28:44.830Z}, providerInstanceId=Optional.absent()}
[2019-02-22 12:41:37.771 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-22 12:41:38.181 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory
[2019-02-22 12:41:38.182 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory
[2019-02-22 12:41:38.282 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: tar: Removing leading `/' from member names
[2019-02-22 12:41:38.690 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Inspecting capabilities of 10.142.15.205
[2019-02-22 12:41:38.716 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-22T12:56:38.716Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.15.205, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.15.205', hostAddress=Optional.of(/10.142.15.205)}, HostEndpoint{hostAddressString='34.73.239.227', hostAddress=Optional.of(/34.73.239.227)}]} Instance{virtualInstance=VirtualInstance{id='56daae7a-fca2-4b2c-ae15-f08806ffdd1e', template=InstanceTemplate{name='centos-6-9', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=10, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(4b1965e7-f35d-4680-898a-8588412fd98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-22T12:28:44.830Z, lastChecked=2019-02-22T12:28:44.830Z}, providerInstanceId=Optional.absent()}
[2019-02-22 12:41:38.720 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-22 12:41:38.722 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.c.ssh.SshClientPoolFactory: Created new ssh client com.cloudera.launchpad.sshj.SshJClient@1c8f3ba4 for endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@5c0d91a0[[10.142.15.205:22, 34.73.239.227:22],SshCredentials{username='ranashd', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-22 12:41:38.755 +0000] INFO  [io-thread-2] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: 97add349)
[2019-02-22 12:41:38.872 +0000] WARN  [reader] - - - - - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.142.15.205 was automatically accepted
[2019-02-22 12:41:38.909 +0000] INFO  [io-thread-10] - - - - - ssh:10.142.15.205: Permissive
[2019-02-22 12:41:39.002 +0000] INFO  [io-thread-9] - - - - - ssh:10.142.15.205: uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[2019-02-22 12:41:39.009 +0000] INFO  [io-thread-1] - - - - - ssh:10.142.15.205: Python 2.6.6
[2019-02-22 12:41:39.220 +0000] INFO  [io-thread-7] - - - - - ssh:10.142.15.205: CentOS release 6.9 (Final)
[2019-02-22 12:41:39.221 +0000] INFO  [io-thread-7] - - - - - c.c.l.i.OperatingSystemCapabilitiesInspector: Using file /etc/centos-release
[2019-02-22 12:41:39.224 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal
[2019-02-22 12:41:39.539 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: Server:		169.254.169.254
[2019-02-22 12:41:39.540 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: Address:	169.254.169.254#53
[2019-02-22 12:41:39.540 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: Non-authoritative answer:
[2019-02-22 12:41:39.540 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: Name:	director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal
[2019-02-22 12:41:39.540 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: Address: 10.142.15.205
[2019-02-22 12:41:39.540 +0000] INFO  [io-thread-6] - - - - - c.c.l.i.OperatingSystemCapabilitiesInspector: nslookup output: Server:		169.254.169.254
Address:	169.254.169.254#53

Non-authoritative answer:
Name:	director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal
Address: 10.142.15.205


[2019-02-22 12:41:39.708 +0000] INFO  [io-thread-3] - - - - - ssh:10.142.15.205: /usr/java/jdk1.8.0_181-cloudera
[2019-02-22 12:41:39.708 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.inspector.util.JdkInspector: Found JAVA_HOME: /usr/java/jdk1.8.0_181-cloudera
[2019-02-22 12:41:40.568 +0000] INFO  [io-thread-2] - - - - - ssh:10.142.15.205: java version "1.8.0_181"
[2019-02-22 12:41:40.568 +0000] INFO  [io-thread-2] - - - - - ssh:10.142.15.205: Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
[2019-02-22 12:41:40.568 +0000] INFO  [io-thread-2] - - - - - ssh:10.142.15.205: Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
[2019-02-22 12:41:42.297 +0000] INFO  [io-thread-2] - - - - - ssh:10.142.15.205: javac 1.8.0_181
[2019-02-22 12:41:43.131 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Running Deployment post create scripts
[2019-02-22 12:41:43.292 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-22 12:41:43.748 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-22 12:41:44.203 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.l.b.c.util.BootstrapClusterUtils: Found Cloudera Manager host ID 060e70ab-e745-47ee-849d-6a0ef6aebc55 for instance with IP 10.142.15.205.
[2019-02-22 12:41:44.203 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.l.b.c.util.BootstrapClusterUtils: Found Cloudera Manager host URL http://director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/060e70ab-e745-47ee-849d-6a0ef6aebc55 for instance with IP 10.142.15.205.
[2019-02-22 12:41:44.507 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments - - c.c.l.p.s.PipelineRepositoryService: Pipeline '11f7036a-b02b-4a12-8c35-87732cc2af68': RUNNING -> COMPLETED
[2019-02-22 12:41:44.595 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments - - c.c.l.d.DeploymentRepositoryService: Deployment 'dep1': BOOTSTRAPPING -> READY
[2019-02-22 12:43:16.029 +0000] INFO  [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:43:46.026 +0000] INFO  [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:44:16.004 +0000] INFO  [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:44:46.012 +0000] INFO  [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:45:16.017 +0000] INFO  [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:45:16.168 +0000] INFO  [task-thread-10] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-22 12:45:16.169 +0000] INFO  [task-thread-10] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-22 12:45:16.195 +0000] INFO  [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2019-02-22 12:45:16.199 +0000] INFO  [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Refreshing deployment env1:dep1.
[2019-02-22 12:45:17.055 +0000] INFO  [task-thread-10] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-22 12:45:17.518 +0000] INFO  [task-thread-10] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-22 12:45:17.760 +0000] ERROR [task-thread-10] - - - - - c.c.l.b.d.util.DeploymentRefresher: Exception while trying to populate CM version.
[2019-02-22 12:45:17.769 +0000] INFO  [task-thread-10] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-22T13:00:17.769Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.15.205', hostEndpoints='[HostEndpoint{hostAddressString='10.142.15.205', hostAddress=Optional.of(/10.142.15.205)}, HostEndpoint{hostAddressString='34.73.239.227', hostAddress=Optional.of(/34.73.239.227)}]', properties={launchTime=Fri Feb 22 12:28:20 UTC 2019, imageId=centos-6-v20180611, instanceId=director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e, publicIpAddress=34.73.239.227, instanceType=n1-standard-1, privateIpAddress=10.142.15.205}} Instance{virtualInstance=VirtualInstance{id='56daae7a-fca2-4b2c-ae15-f08806ffdd1e', template=InstanceTemplate{name='centos-6-9', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=10, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(4b1965e7-f35d-4680-898a-8588412fd98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(060e70ab-e745-47ee-849d-6a0ef6aebc55), cmHostUrl=Optional.of(http://director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/060e70ab-e745-47ee-849d-6a0ef6aebc55), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-22T12:41:43.962Z, lastChecked=2019-02-22T12:41:43.962Z}, providerInstanceId=Optional.absent()}
[2019-02-22 12:45:17.770 +0000] INFO  [task-thread-10] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-22 12:45:27.915 +0000] INFO  [io-thread-1] - - - - - ssh:10.142.15.205: https://archive.cloudera.com/cm6/6.1.1/redhat6/yum/RPMS/x86_64/cloudera-manager-server-6.1.1-853290.el6.x86_64.rpm
[2019-02-22 12:45:28.234 +0000] INFO  [task-thread-10] - - - - - c.c.l.b.d.util.DeploymentRefresher: Detected a change in the CM Server Package, setting the CM Repository URL to https://archive.cloudera.com/cm6/6.1.1/redhat6/yum
[2019-02-22 12:45:28.234 +0000] INFO  [task-thread-10] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-22T13:00:28.234Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.15.205', hostEndpoints='[HostEndpoint{hostAddressString='10.142.15.205', hostAddress=Optional.of(/10.142.15.205)}, HostEndpoint{hostAddressString='34.73.239.227', hostAddress=Optional.of(/34.73.239.227)}]', properties={launchTime=Fri Feb 22 12:28:20 UTC 2019, imageId=centos-6-v20180611, instanceId=director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e, publicIpAddress=34.73.239.227, instanceType=n1-standard-1, privateIpAddress=10.142.15.205}} Instance{virtualInstance=VirtualInstance{id='56daae7a-fca2-4b2c-ae15-f08806ffdd1e', template=InstanceTemplate{name='centos-6-9', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=10, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(4b1965e7-f35d-4680-898a-8588412fd98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(060e70ab-e745-47ee-849d-6a0ef6aebc55), cmHostUrl=Optional.of(http://director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/060e70ab-e745-47ee-849d-6a0ef6aebc55), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-22T12:41:43.962Z, lastChecked=2019-02-22T12:41:43.962Z}, providerInstanceId=Optional.absent()}
[2019-02-22 12:45:28.236 +0000] INFO  [task-thread-10] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-22 12:45:28.541 +0000] INFO  [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Backing up Cloudera Manager configuration for Deployment env1:dep1
[2019-02-22 12:45:28.548 +0000] INFO  [task-thread-10] - - - - - c.c.l.p.DatabasePipelineService: Starting pipeline '6c7adf4b-3c06-4fd8-a0b2-ede485c22a84' with root job com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2019-02-22 12:45:28.569 +0000] INFO  [task-thread-10] - - - - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '6c7adf4b-3c06-4fd8-a0b2-ede485c22a84'
[2019-02-22 12:45:28.600 +0000] INFO  [p-ede485c22a84-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-22T13:00:28.600Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.15.205, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.15.205', hostAddress=Optional.of(/10.142.15.205)}, HostEndpoint{hostAddressString='34.73.239.227', hostAddress=Optional.of(/34.73.239.227)}]} Instance{virtualInstance=VirtualInstance{id='56daae7a-fca2-4b2c-ae15-f08806ffdd1e', template=InstanceTemplate{name='centos-6-9', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=10, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(4b1965e7-f35d-4680-898a-8588412fd98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(060e70ab-e745-47ee-849d-6a0ef6aebc55), cmHostUrl=Optional.of(http://director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/060e70ab-e745-47ee-849d-6a0ef6aebc55), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-22T12:45:17.731Z, lastChecked=2019-02-22T12:45:17.731Z}, providerInstanceId=Optional.absent()}
[2019-02-22 12:45:28.601 +0000] INFO  [p-ede485c22a84-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-22 12:45:29.040 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory
[2019-02-22 12:45:29.041 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory
[2019-02-22 12:45:29.241 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: tar: Removing leading `/' from member names
[2019-02-22 12:45:29.761 +0000] INFO  [p-ede485c22a84-BackupClouderaManagerConfig] - - - - - c.c.l.p.s.PipelineRepositoryService: Pipeline '6c7adf4b-3c06-4fd8-a0b2-ede485c22a84': RUNNING -> COMPLETED
[2019-02-22 12:45:29.953 +0000] INFO  [io-thread-6] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: cf364410)
[2019-02-22 12:45:30.082 +0000] INFO  [task-thread-10] - - - - - c.c.l.p.DatabasePipelineService: Deleting pipeline '6c7adf4b-3c06-4fd8-a0b2-ede485c22a84'
[2019-02-22 12:45:30.229 +0000] INFO  [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models
[2019-02-22 12:45:46.015 +0000] INFO  [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:46:16.009 +0000] INFO  [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:46:16.056 +0000] INFO  [task-thread-5] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-22 12:46:16.063 +0000] INFO  [task-thread-5] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-22 12:46:46.040 +0000] INFO  [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:47:16.010 +0000] INFO  [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:47:46.005 +0000] INFO  [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:48:16.007 +0000] INFO  [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:48:46.010 +0000] INFO  [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null

[#2]

This error is benign -- it won't affect the behavior of your clusters -- but I'm a little curious as to why it's happening. Did you change the admin credentials in Cloudera Manager, by chance?

[#3]

Hi Mike,

After some investigation found out that it all started while creating the "deployment". This error says your "The deployment may have failed or may have a connectivity issue.". Both nodes 'director' and 'cm-manager' can ping, even i tried entries in /etc/hosts file.  Now i am investing may be I am not using the SSH keys propperly. 

I am deploying on GCP, and using same ssh-key to ssh from putty. This key is fine for both hosts. but looks while communication between 'director' and 'cm-maanger' using ssh key looks issue. Any thoughts to trouble shoot.

i am using the default network offered in GCP.  Further 'cm-manager' nodes which is the one which is created while creatign the deployment got a network tag "cloudera-director-google-plugin-2-0-1" . i am not sure about this.

Thanks,

Rana

[#4]

I'm also getting the same error. Please let me know if you were able to resolve this error. Thanks.

[#5]

I sorted this out by installing the jdbc drivers mentioned in the documentation.

wish you best luck 🙂

[#6]

Thanks Rana. Are you actively using GCP cluster. I am able to resolve the
issue by just redoing it and proceeding further.

Wanted to know abt your experience.

Thanks

[#7]

Hi man,

My recommendations would be to go with GCP, these days i am prepare for GCP exam.

Thanks,

Rana