Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Cannot ping CM DeploymentKey

Cannot ping CM DeploymentKey

Explorer

Hi 

 

i setup the director on GCP and then added the environment using webui and created the deployment which resulted the error at the end cann't ping the CM deployment key. if i continue and add cluster all the nodes are provised but this error keeps on comming back. 

 

I am able to ssh from my local network to GCP and even able too ssh from director node to cloudera maanger

node using the ssh key provided while creating the environment. i am badly stuck. Here are lat line from the log.

2019-02-22 12:39:35.546 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.launchpad.cm.ApiCommandMonitor: Waiting for commands: [14]
[2019-02-22 12:40:16.167 +0000] INFO  [task-thread-7] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-22 12:40:16.168 +0000] INFO  [task-thread-7] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-22 12:40:16.190 +0000] INFO  [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2019-02-22 12:40:16.191 +0000] INFO  [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Skipping refresh of deployment env1:dep1 as it is in transition or terminated. (Stage: BOOTSTRAPPING, deploymentIsNull: true)
[2019-02-22 12:40:16.191 +0000] INFO  [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models
[2019-02-22 12:41:16.049 +0000] INFO  [task-thread-2] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-22 12:41:16.055 +0000] INFO  [task-thread-2] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-22 12:41:21.857 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.b.c.BootstrapClouderaManagerAgent: Command GlobalHostInstall with ID 14 completed successfully on instance 10.142.15.205. Details: ApiCommand{id=14, name=GlobalHostInstall, startTime=Fri Feb 22 12:39:34 UTC 2019, endTime=Fri Feb 22 12:41:12 UTC 2019, active=false, success=true, resultMessage=Successfully completed installation., serviceRef=null, roleRef=null, hostRef=null, parent=null}
[2019-02-22 12:41:27.841 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Setting up Cloudera Management Services
[2019-02-22 12:41:28.502 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.SetupClouderaManagementServices$CreateMgmtService - c.c.l.b.d.SetupClouderaManagementServices: Auto assigning management service roles
[2019-02-22 12:41:37.621 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.ConfigureClouderaManagementServices - c.c.l.b.d.ConfigureClouderaManagementServices: Restarting Cloudera Management Services: ApiCommand{id=24, name=Restart, startTime=Fri Feb 22 12:41:34 UTC 2019, endTime=null, active=true, success=null, resultMessage=null, serviceRef=ApiServiceRef{peerName=null, clusterName=null, serviceName=MGMT}, roleRef=null, hostRef=null, parent=null}
[2019-02-22 12:41:37.633 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Backing up Cloudera Manager Server configuration
[2019-02-22 12:41:37.767 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-22T12:56:37.767Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.15.205, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.15.205', hostAddress=Optional.of(/10.142.15.205)}, HostEndpoint{hostAddressString='34.73.239.227', hostAddress=Optional.of(/34.73.239.227)}]} Instance{virtualInstance=VirtualInstance{id='56daae7a-fca2-4b2c-ae15-f08806ffdd1e', template=InstanceTemplate{name='centos-6-9', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=10, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(4b1965e7-f35d-4680-898a-8588412fd98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-22T12:28:44.830Z, lastChecked=2019-02-22T12:28:44.830Z}, providerInstanceId=Optional.absent()}
[2019-02-22 12:41:37.771 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-22 12:41:38.181 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory
[2019-02-22 12:41:38.182 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory
[2019-02-22 12:41:38.282 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: tar: Removing leading `/' from member names
[2019-02-22 12:41:38.690 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Inspecting capabilities of 10.142.15.205
[2019-02-22 12:41:38.716 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-22T12:56:38.716Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.15.205, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.15.205', hostAddress=Optional.of(/10.142.15.205)}, HostEndpoint{hostAddressString='34.73.239.227', hostAddress=Optional.of(/34.73.239.227)}]} Instance{virtualInstance=VirtualInstance{id='56daae7a-fca2-4b2c-ae15-f08806ffdd1e', template=InstanceTemplate{name='centos-6-9', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=10, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(4b1965e7-f35d-4680-898a-8588412fd98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-22T12:28:44.830Z, lastChecked=2019-02-22T12:28:44.830Z}, providerInstanceId=Optional.absent()}
[2019-02-22 12:41:38.720 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-22 12:41:38.722 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.c.ssh.SshClientPoolFactory: Created new ssh client com.cloudera.launchpad.sshj.SshJClient@1c8f3ba4 for endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@5c0d91a0[[10.142.15.205:22, 34.73.239.227:22],SshCredentials{username='ranashd', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-22 12:41:38.755 +0000] INFO  [io-thread-2] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: 97add349)
[2019-02-22 12:41:38.872 +0000] WARN  [reader] - - - - - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.142.15.205 was automatically accepted
[2019-02-22 12:41:38.909 +0000] INFO  [io-thread-10] - - - - - ssh:10.142.15.205: Permissive
[2019-02-22 12:41:39.002 +0000] INFO  [io-thread-9] - - - - - ssh:10.142.15.205: uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[2019-02-22 12:41:39.009 +0000] INFO  [io-thread-1] - - - - - ssh:10.142.15.205: Python 2.6.6
[2019-02-22 12:41:39.220 +0000] INFO  [io-thread-7] - - - - - ssh:10.142.15.205: CentOS release 6.9 (Final)
[2019-02-22 12:41:39.221 +0000] INFO  [io-thread-7] - - - - - c.c.l.i.OperatingSystemCapabilitiesInspector: Using file /etc/centos-release
[2019-02-22 12:41:39.224 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal
[2019-02-22 12:41:39.539 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: Server:		169.254.169.254
[2019-02-22 12:41:39.540 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: Address:	169.254.169.254#53
[2019-02-22 12:41:39.540 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: Non-authoritative answer:
[2019-02-22 12:41:39.540 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: Name:	director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal
[2019-02-22 12:41:39.540 +0000] INFO  [io-thread-6] - - - - - ssh:10.142.15.205: Address: 10.142.15.205
[2019-02-22 12:41:39.540 +0000] INFO  [io-thread-6] - - - - - c.c.l.i.OperatingSystemCapabilitiesInspector: nslookup output: Server:		169.254.169.254
Address:	169.254.169.254#53

Non-authoritative answer:
Name:	director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal
Address: 10.142.15.205


[2019-02-22 12:41:39.708 +0000] INFO  [io-thread-3] - - - - - ssh:10.142.15.205: /usr/java/jdk1.8.0_181-cloudera
[2019-02-22 12:41:39.708 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.inspector.util.JdkInspector: Found JAVA_HOME: /usr/java/jdk1.8.0_181-cloudera
[2019-02-22 12:41:40.568 +0000] INFO  [io-thread-2] - - - - - ssh:10.142.15.205: java version "1.8.0_181"
[2019-02-22 12:41:40.568 +0000] INFO  [io-thread-2] - - - - - ssh:10.142.15.205: Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
[2019-02-22 12:41:40.568 +0000] INFO  [io-thread-2] - - - - - ssh:10.142.15.205: Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
[2019-02-22 12:41:42.297 +0000] INFO  [io-thread-2] - - - - - ssh:10.142.15.205: javac 1.8.0_181
[2019-02-22 12:41:43.131 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Running Deployment post create scripts
[2019-02-22 12:41:43.292 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-22 12:41:43.748 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-22 12:41:44.203 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.l.b.c.util.BootstrapClusterUtils: Found Cloudera Manager host ID 060e70ab-e745-47ee-849d-6a0ef6aebc55 for instance with IP 10.142.15.205.
[2019-02-22 12:41:44.203 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.l.b.c.util.BootstrapClusterUtils: Found Cloudera Manager host URL http://director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/060e70ab-e745-47ee-849d-6a0ef6aebc55 for instance with IP 10.142.15.205.
[2019-02-22 12:41:44.507 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments - - c.c.l.p.s.PipelineRepositoryService: Pipeline '11f7036a-b02b-4a12-8c35-87732cc2af68': RUNNING -> COMPLETED
[2019-02-22 12:41:44.595 +0000] INFO  [p-87732cc2af68-DefaultBootstrapDeploymentJob] 2a0548ab-8c9f-487a-bf74-ac6dc59ae6e6 POST /api/d6.1/environments/env1/deployments - - c.c.l.d.DeploymentRepositoryService: Deployment 'dep1': BOOTSTRAPPING -> READY
[2019-02-22 12:43:16.029 +0000] INFO  [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:43:46.026 +0000] INFO  [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:44:16.004 +0000] INFO  [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:44:46.012 +0000] INFO  [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:45:16.017 +0000] INFO  [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:45:16.168 +0000] INFO  [task-thread-10] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-22 12:45:16.169 +0000] INFO  [task-thread-10] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-22 12:45:16.195 +0000] INFO  [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2019-02-22 12:45:16.199 +0000] INFO  [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Refreshing deployment env1:dep1.
[2019-02-22 12:45:17.055 +0000] INFO  [task-thread-10] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-22 12:45:17.518 +0000] INFO  [task-thread-10] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-22 12:45:17.760 +0000] ERROR [task-thread-10] - - - - - c.c.l.b.d.util.DeploymentRefresher: Exception while trying to populate CM version.
[2019-02-22 12:45:17.769 +0000] INFO  [task-thread-10] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-22T13:00:17.769Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.15.205', hostEndpoints='[HostEndpoint{hostAddressString='10.142.15.205', hostAddress=Optional.of(/10.142.15.205)}, HostEndpoint{hostAddressString='34.73.239.227', hostAddress=Optional.of(/34.73.239.227)}]', properties={launchTime=Fri Feb 22 12:28:20 UTC 2019, imageId=centos-6-v20180611, instanceId=director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e, publicIpAddress=34.73.239.227, instanceType=n1-standard-1, privateIpAddress=10.142.15.205}} Instance{virtualInstance=VirtualInstance{id='56daae7a-fca2-4b2c-ae15-f08806ffdd1e', template=InstanceTemplate{name='centos-6-9', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=10, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(4b1965e7-f35d-4680-898a-8588412fd98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(060e70ab-e745-47ee-849d-6a0ef6aebc55), cmHostUrl=Optional.of(http://director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/060e70ab-e745-47ee-849d-6a0ef6aebc55), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-22T12:41:43.962Z, lastChecked=2019-02-22T12:41:43.962Z}, providerInstanceId=Optional.absent()}
[2019-02-22 12:45:17.770 +0000] INFO  [task-thread-10] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-22 12:45:27.915 +0000] INFO  [io-thread-1] - - - - - ssh:10.142.15.205: https://archive.cloudera.com/cm6/6.1.1/redhat6/yum/RPMS/x86_64/cloudera-manager-server-6.1.1-853290.el6.x86_64.rpm
[2019-02-22 12:45:28.234 +0000] INFO  [task-thread-10] - - - - - c.c.l.b.d.util.DeploymentRefresher: Detected a change in the CM Server Package, setting the CM Repository URL to https://archive.cloudera.com/cm6/6.1.1/redhat6/yum
[2019-02-22 12:45:28.234 +0000] INFO  [task-thread-10] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-22T13:00:28.234Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.15.205', hostEndpoints='[HostEndpoint{hostAddressString='10.142.15.205', hostAddress=Optional.of(/10.142.15.205)}, HostEndpoint{hostAddressString='34.73.239.227', hostAddress=Optional.of(/34.73.239.227)}]', properties={launchTime=Fri Feb 22 12:28:20 UTC 2019, imageId=centos-6-v20180611, instanceId=director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e, publicIpAddress=34.73.239.227, instanceType=n1-standard-1, privateIpAddress=10.142.15.205}} Instance{virtualInstance=VirtualInstance{id='56daae7a-fca2-4b2c-ae15-f08806ffdd1e', template=InstanceTemplate{name='centos-6-9', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=10, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(4b1965e7-f35d-4680-898a-8588412fd98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(060e70ab-e745-47ee-849d-6a0ef6aebc55), cmHostUrl=Optional.of(http://director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/060e70ab-e745-47ee-849d-6a0ef6aebc55), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-22T12:41:43.962Z, lastChecked=2019-02-22T12:41:43.962Z}, providerInstanceId=Optional.absent()}
[2019-02-22 12:45:28.236 +0000] INFO  [task-thread-10] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-22 12:45:28.541 +0000] INFO  [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Backing up Cloudera Manager configuration for Deployment env1:dep1
[2019-02-22 12:45:28.548 +0000] INFO  [task-thread-10] - - - - - c.c.l.p.DatabasePipelineService: Starting pipeline '6c7adf4b-3c06-4fd8-a0b2-ede485c22a84' with root job com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2019-02-22 12:45:28.569 +0000] INFO  [task-thread-10] - - - - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '6c7adf4b-3c06-4fd8-a0b2-ede485c22a84'
[2019-02-22 12:45:28.600 +0000] INFO  [p-ede485c22a84-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-22T13:00:28.600Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.15.205, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.15.205', hostAddress=Optional.of(/10.142.15.205)}, HostEndpoint{hostAddressString='34.73.239.227', hostAddress=Optional.of(/34.73.239.227)}]} Instance{virtualInstance=VirtualInstance{id='56daae7a-fca2-4b2c-ae15-f08806ffdd1e', template=InstanceTemplate{name='centos-6-9', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=10, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(4b1965e7-f35d-4680-898a-8588412fd98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(060e70ab-e745-47ee-849d-6a0ef6aebc55), cmHostUrl=Optional.of(http://director-56daae7a-fca2-4b2c-ae15-f08806ffdd1e.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/060e70ab-e745-47ee-849d-6a0ef6aebc55), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-22T12:45:17.731Z, lastChecked=2019-02-22T12:45:17.731Z}, providerInstanceId=Optional.absent()}
[2019-02-22 12:45:28.601 +0000] INFO  [p-ede485c22a84-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-22 12:45:29.040 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory
[2019-02-22 12:45:29.041 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory
[2019-02-22 12:45:29.241 +0000] INFO  [io-thread-8] - - - - - ssh:10.142.15.205: tar: Removing leading `/' from member names
[2019-02-22 12:45:29.761 +0000] INFO  [p-ede485c22a84-BackupClouderaManagerConfig] - - - - - c.c.l.p.s.PipelineRepositoryService: Pipeline '6c7adf4b-3c06-4fd8-a0b2-ede485c22a84': RUNNING -> COMPLETED
[2019-02-22 12:45:29.953 +0000] INFO  [io-thread-6] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: cf364410)
[2019-02-22 12:45:30.082 +0000] INFO  [task-thread-10] - - - - - c.c.l.p.DatabasePipelineService: Deleting pipeline '6c7adf4b-3c06-4fd8-a0b2-ede485c22a84'
[2019-02-22 12:45:30.229 +0000] INFO  [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models
[2019-02-22 12:45:46.015 +0000] INFO  [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:46:16.009 +0000] INFO  [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:46:16.056 +0000] INFO  [task-thread-5] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-22 12:46:16.063 +0000] INFO  [task-thread-5] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-22 12:46:46.040 +0000] INFO  [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:47:16.010 +0000] INFO  [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:47:46.005 +0000] INFO  [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:48:16.007 +0000] INFO  [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-22 12:48:46.010 +0000] INFO  [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null

 

6 REPLIES 6

Re: Cannot ping CM DeploymentKey

Expert Contributor
This error is benign -- it won't affect the behavior of your clusters -- but I'm a little curious as to why it's happening. Did you change the admin credentials in Cloudera Manager, by chance?

Re: Cannot ping CM DeploymentKey

Explorer

Hi Mike,

 

After some investigation found out that it all started while creating the "deployment". This error says your "The deployment may have failed or may have a connectivity issue.". Both nodes 'director' and 'cm-manager' can ping, even i tried entries in /etc/hosts file.  Now i am investing may be I am not using the SSH keys propperly. 

I am deploying on GCP, and using same ssh-key to ssh from putty. This key is fine for both hosts. but looks while communication between 'director' and 'cm-maanger' using ssh key looks issue. Any thoughts to trouble shoot.

 

i am using the default network offered in GCP.  Further 'cm-manager' nodes which is the one which is created while creatign the deployment got a network tag "cloudera-director-google-plugin-2-0-1" . i am not sure about this.

 

Thanks,

Rana

 

 

Highlighted

Re: Cannot ping CM DeploymentKey

New Contributor

I'm also getting the same error. Please let me know if you were able to resolve this error. Thanks.

Re: Cannot ping CM DeploymentKey

Explorer

I sorted this out by installing the jdbc drivers mentioned in the documentation.

 

wish you best luck :)

Re: Cannot ping CM DeploymentKey

New Contributor

Thanks Rana. Are you actively using GCP cluster. I am able to resolve the
issue by just redoing it and proceeding further.

Wanted to know abt your experience.

Thanks




Re: Cannot ping CM DeploymentKey

Explorer

Hi man,

 

My recommendations would be to go with GCP, these days i am prepare for GCP exam.

 

Thanks,

Rana