Support Questions
Find answers, ask questions, and share your expertise
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- Cloudera Community
- Support
- Support Questions
- Opening `direct-tcpip` channel failed: Connection ...
Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Opening `direct-tcpip` channel failed: Connection refused
Labels:
- Labels:
-
Cloudera Manager
Contributor
Created on 02-16-2019 10:47 PM - edited 09-16-2022 08:51 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All,
just doing first installation on GCP, stuck with this error, wellcome to any suggestions. Thanks
[2019-02-17 10:21:28.847 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Waiting for Cloudera Manager server to start
[2019-02-17 10:21:28.860 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.l.b.WaitForSshTunnelToServerOnPort: Waiting for tunnel to server 10.142.0.59 on port 7180 until 2019-02-17T10:31:23.635+05:00
[2019-02-17 10:21:28.860 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-17T10:36:28.860+05:00 to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.59, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.59', hostAddress=Optional.of(/10.142.0.59)}, HostEndpoint{hostAddressString='34.73.227.81', hostAddress=Optional.of(/34.73.227.81)}]} Instance{virtualInstance=VirtualInstance{id='4aeb5b1e-2837-4c19-b049-78c22c3a39f8', template=InstanceTemplate{name='centos7', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=50, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=50}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(846f1bcc-69e7-408e-a090-5006f885821b)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-4aeb5b1e-2837-4c19-b049-78c22c3a39f8.us-east1-b.c.cloudera-cluster-project.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-17T10:13:38.083+05:00, lastChecked=2019-02-17T10:13:38.083+05:00}, providerInstanceId=Optional.absent()}
[2019-02-17 10:21:28.860 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-17 10:21:28.894 +0500] ERROR [port-forwarding-[40111:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#46 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:21:38.899 +0500] ERROR [port-forwarding-[34772:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#47 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:21:48.904 +0500] ERROR [port-forwarding-[45887:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#48 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:21:58.909 +0500] ERROR [port-forwarding-[38046:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#49 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:22:08.913 +0500] ERROR [port-forwarding-[44222:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#50 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:22:18.915 +0500] ERROR [port-forwarding-[35107:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#51 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:22:28.919 +0500] ERROR [port-forwarding-[39019:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#52 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:22:37.424 +0500] INFO [task-thread-4] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-17 10:22:37.437 +0500] INFO [task-thread-4] - - - - - c.c.launchpad.task.RefreshClusters: Skipping refresh of cluster env-1:dep1:cluster1 as it is in transition, terminated, or has failed updating. (Stage: BOOTSTRAPPING, clusterIsNull: true)
[2019-02-17 10:22:37.437 +0500] INFO [task-thread-4] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-17 10:22:38.924 +0500] ERROR [port-forwarding-[42425:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#53 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:22:48.928 +0500] ERROR [port-forwarding-[35073:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#54 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:22:58.937 +0500] ERROR [port-forwarding-[32979:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#55 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:23:08.942 +0500] ERROR [port-forwarding-[39740:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#56 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:23:18.946 +0500] ERROR [port-forwarding-[34086:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#57 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:23:28.949 +0500] ERROR [port-forwarding-[36147:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#58 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:23:36.583 +0500] INFO [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2019-02-17 10:23:36.583 +0500] INFO [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Skipping refresh of deployment env-1:dep1 as it is in transition or terminated. (Stage: BOOTSTRAPPING, deploymentIsNull: true)
[2019-02-17 10:23:36.583 +0500] INFO [task-thread-10] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models
[2019-02-17 10:23:37.514 +0500] INFO [task-thread-10] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-17 10:23:37.517 +0500] INFO [task-thread-10] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-17 10:23:38.953 +0500] ERROR [port-forwarding-[44393:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#59 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:23:48.956 +0500] ERROR [port-forwarding-[40884:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#60 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:23:58.959 +0500] ERROR [port-forwarding-[36902:10.142.0.59:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#61 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-17 10:24:08.965 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.l.b.WaitForSshTunnelToServerOnPort: Successfully established tunnel to server 10.142.0.59 at 7180
[2019-02-17 10:25:58.716 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-17T10:40:58.716+05:00 to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.59, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.59', hostAddress=Optional.of(/10.142.0.59)}, HostEndpoint{hostAddressString='34.73.227.81', hostAddress=Optional.of(/34.73.227.81)}]} Instance{virtualInstance=VirtualInstance{id='4aeb5b1e-2837-4c19-b049-78c22c3a39f8', template=InstanceTemplate{name='centos7', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=50, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=50}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(846f1bcc-69e7-408e-a090-5006f885821b)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-4aeb5b1e-2837-4c19-b049-78c22c3a39f8.us-east1-b.c.cloudera-cluster-project.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-17T10:13:38.083+05:00, lastChecked=2019-02-17T10:13:38.083+05:00}, providerInstanceId=Optional.absent()}
[2019-02-17 10:25:58.716 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-17 10:25:59.379 +0500] INFO [io-thread-4] - - - - - ssh:34.73.227.81: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory
[2019-02-17 10:25:59.379 +0500] INFO [io-thread-4] - - - - - ssh:34.73.227.81: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory
[2019-02-17 10:25:59.480 +0500] INFO [io-thread-4] - - - - - ssh:34.73.227.81: tar: Removing leading `/' from member names
[2019-02-17 10:25:59.743 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Inspecting capabilities of 10.142.0.59
[2019-02-17 10:25:59.752 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-17T10:40:59.752+05:00 to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.59, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.59', hostAddress=Optional.of(/10.142.0.59)}, HostEndpoint{hostAddressString='34.73.227.81', hostAddress=Optional.of(/34.73.227.81)}]} Instance{virtualInstance=VirtualInstance{id='4aeb5b1e-2837-4c19-b049-78c22c3a39f8', template=InstanceTemplate{name='centos7', type='n1-standard-1', image='centos6', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=50, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=50}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.of(ranashd), sshHostKeyRetrievalType=NONE, groupId=Optional.of(846f1bcc-69e7-408e-a090-5006f885821b)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_6, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.6.6), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-4aeb5b1e-2837-4c19-b049-78c22c3a39f8.us-east1-b.c.cloudera-cluster-project.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-17T10:13:38.083+05:00, lastChecked=2019-02-17T10:13:38.083+05:00}, providerInstanceId=Optional.absent()}
[2019-02-17 10:25:59.753 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-17 10:25:59.753 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.c.ssh.SshClientPoolFactory: Created new ssh client com.cloudera.launchpad.sshj.SshJClient@49a97b1b for endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@57a42df[[10.142.0.59:22, 34.73.227.81:22],SshCredentials{username='ranashd', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-17 10:25:59.912 +0500] WARN [reader] - - - - - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 34.73.227.81 was automatically accepted
[2019-02-17 10:25:59.925 +0500] INFO [io-thread-3] - - - - - ssh:34.73.227.81: CentOS release 6.9 (Final)
[2019-02-17 10:25:59.925 +0500] INFO [io-thread-3] - - - - - c.c.l.i.OperatingSystemCapabilitiesInspector: Using file /etc/centos-release
[2019-02-17 10:25:59.928 +0500] INFO [io-thread-1] - - - - - ssh:34.73.227.81: Python 2.6.6
[2019-02-17 10:25:59.931 +0500] INFO [io-thread-8] - - - - - ssh:34.73.227.81: Permissive
[2019-02-17 10:25:59.966 +0500] INFO [io-thread-6] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: 12875d15)
[2019-02-17 10:25:59.977 +0500] INFO [io-thread-2] - - - - - ssh:34.73.227.81: uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[2019-02-17 10:26:00.209 +0500] INFO [io-thread-5] - - - - - ssh:34.73.227.81: director-4aeb5b1e-2837-4c19-b049-78c22c3a39f8.us-east1-b.c.cloudera-cluster-project.internal
[2019-02-17 10:26:00.477 +0500] INFO [io-thread-7] - - - - - ssh:34.73.227.81: /usr/java/jdk1.8.0_141-cloudera
[2019-02-17 10:26:00.478 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.inspector.util.JdkInspector: Found JAVA_HOME: /usr/java/jdk1.8.0_141-cloudera
[2019-02-17 10:26:00.634 +0500] INFO [io-thread-5] - - - - - ssh:34.73.227.81: Server: 169.254.169.254
[2019-02-17 10:26:00.634 +0500] INFO [io-thread-5] - - - - - ssh:34.73.227.81: Address: 169.254.169.254#53
[2019-02-17 10:26:00.634 +0500] INFO [io-thread-5] - - - - - ssh:34.73.227.81: Non-authoritative answer:
[2019-02-17 10:26:00.634 +0500] INFO [io-thread-5] - - - - - ssh:34.73.227.81: Name: director-4aeb5b1e-2837-4c19-b049-78c22c3a39f8.us-east1-b.c.cloudera-cluster-project.internal
[2019-02-17 10:26:00.634 +0500] INFO [io-thread-5] - - - - - ssh:34.73.227.81: Address: 10.142.0.59
[2019-02-17 10:26:00.634 +0500] INFO [io-thread-5] - - - - - c.c.l.i.OperatingSystemCapabilitiesInspector: nslookup output: Server: 169.254.169.254
Address: 169.254.169.254#53
Non-authoritative answer:
Name: director-4aeb5b1e-2837-4c19-b049-78c22c3a39f8.us-east1-b.c.cloudera-cluster-project.internal
Address: 10.142.0.59
[2019-02-17 10:26:01.375 +0500] INFO [io-thread-3] - - - - - ssh:34.73.227.81: java version "1.8.0_141"
[2019-02-17 10:26:01.375 +0500] INFO [io-thread-3] - - - - - ssh:34.73.227.81: Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
[2019-02-17 10:26:01.375 +0500] INFO [io-thread-3] - - - - - ssh:34.73.227.81: Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)
[2019-02-17 10:26:03.649 +0500] INFO [io-thread-3] - - - - - ssh:34.73.227.81: javac 1.8.0_141
[2019-02-17 10:26:04.706 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Running Deployment post create scripts
[2019-02-17 10:26:04.740 +0500] INFO [p-aa8765999b40-DefaultBootstrapDeploymentJob] 48f49890-ecc0-442e-a79e-26bfb8356bbf POST /api/d6.1/environments/env-1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
ef2e61-BootstrapClouderaManagerAgent] 87f387f6-efdc-4404-a7be-28b863d7020d POST /api/d6.1/environments/env-1/deployments/dep1/clusters com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.launchpad.cm.ApiCommandMonitor: Waiting for commands: [38]
[2019-02-17 10:26:19.416 +0500] INFO [p-d47fb76e7b16-BootstrapClouderaManagerAgent] 87f387f6-efdc-4404-a7be-28b863d7020d POST /api/d6.1/environments/env-1/deployments/dep1/clusters com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Waiting for Cloudera Manager to deploy agent on 10.142.0.63
[2019-02-17 10:26:19.446 +0500] INFO [p-d47fb76e7b16-BootstrapClouderaManagerAgent] 87f387f6-efdc-4404-a7be-28b863d7020d POST /api/d6.1/environments/env-1/deployments/dep1/clusters com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.launchpad.cm.ApiCommandMonitor: Waiting for commands: [39]
[2019-02-17 10:27:37.437 +0500] INFO [task-thread-5] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-17 10:27:37.451 +0500] INFO [task-thread-5] - - - - - c.c.launchpad.task.RefreshClusters: Skipping refresh of cluster env-1:dep1:cluster1 as it is in transition, terminated, or has failed updating. (Stage: BOOTSTRAPPING, clusterIsNull: true)
[2019-02-17 10:27:37.451 +0500] INFO [task-thread-5] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-17 10:27:42.560 +0500] ERROR [p-66c81a04d10f-BootstrapClouderaManagerAgent] 87f387f6-efdc-4404-a7be-28b863d7020d POST /api/d6.1/environments/env-1/deployments/dep1/clusters com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
javax.ws.rs.ProcessingException: java.net.ConnectException: ConnectException invoking http://10.142.0.59:7180/api/v12/commands/35: Connection refused (Connection refused)
at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:638)
at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:614)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:791)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:238)
at com.sun.proxy.$Proxy249.readCommand(Unknown Source)
at sun.reflect.GeneratedMethodAccessor499.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:117)
at com.sun.proxy.$Proxy249.readCommand(Unknown Source)
at com.cloudera.launchpad.cm.ApiCommandMonitor.waitUntilInactive(ApiCommandMonitor.java:171)
at com.cloudera.launchpad.cm.ApiCommandMonitor.waitUntilInactive(ApiCommandMonitor.java:144)
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:640)
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:592)
at com.cloudera.launchpad.pipeline.job.Job6.runUnchecked(Job6.java:36)
at com.cloudera.launchpad.pipeline.job.Job6$$FastClassBySpringCGLIB$$54178506.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure$$EnhancerBySpringCGLIB$$fd4076e6.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:202)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:173)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:136)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:201)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:141)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: ConnectException invoking http://10.142.0.59:7180/api/v12/commands/35: Connection refused (Connection refused)
3 REPLIES 3
Super Collaborator
Created 02-19-2019 08:32 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Rana,
The errors for "Opening `direct-tcpip` channel failed: Connection refused" happen under normal circumstances and can be ignored. I thought we'd suppressed them by now in Director logging, but maybe not. 🙂
Still, I see there's a failure to connect to Cloudera Manager at the end of the log snippet. That means that Director was trying to check on the status of a running command and couldn't reach Cloudera Manager. The IP address for that instance of Cloudera Manager is present in prior logging lines, so apparently the instance is at least reachable. Director did have to establish an SSH tunnel to talk to it:
Successfully established tunnel to server 10.142.0.59 at 7180
There's not enough context in the logs here for me to go much further. The pipeline thread for the Cloudera Manager failure, "p-66c81a04d10f-BootstrapClouderaManagerAgent", doesn't appear elsewhere in the sample, but other threads are working with the instance successfully.
Also, the beginning of line 68 is cut off, so I'm concerned that multiple Director instances are running at the same time.
General ideas to troubleshoot:
- Run Director inside GCP so that it can make direct connections to all of the new instances. Once that's working, try running Director outside of it, which is where I think it's running right now.
- At first, just do one thing at a time in Director. It makes troubleshooting the logs easier. (We're actually working on a way to split logging out by cluster to help here.)
- If that instance is still up, see if Cloudera Manager is in fact running. Maybe it died for some reason? Logs in /var/log/cloudera-scm-server are usually informative.
Bill
Contributor
Created 02-19-2019 09:21 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Bill
i first errors are started while creating the deployment. "CM deployment key looks issue here"
[2019-02-19 12:21:00.467 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.EnableTelemetryServices - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:21:00.569 +0000] INFO [io-thread-9] - - - - - ssh:34.73.121.164: export CMF_FF_CME=true
[2019-02-19 12:21:00.586 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager$StartClouderaManager - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:36:00.586Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=true, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:16:05.847Z, lastChecked=2019-02-19T12:16:05.847Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:21:00.586 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager$StartClouderaManager - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:21:00.752 +0000] ERROR [port-forwarding-[46778:10.142.0.42:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#43 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-19 12:21:02.839 +0000] INFO [task-thread-6] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-19 12:21:02.842 +0000] INFO [task-thread-6] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-19 12:21:10.759 +0000] ERROR [port-forwarding-[33141:10.142.0.42:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#44 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2019-02-19 12:23:50.848 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.l.b.WaitForSshTunnelToServerOnPort: Successfully established tunnel to server 10.142.0.42 at 7180
[2019-02-19 12:23:50.876 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Waiting until 2019-02-19T12:23:50.866Z for an accessible port on endpoints [10.142.0.42:7180]
[2019-02-19 12:23:50.876 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Checking every 10000 milliseconds
[2019-02-19 12:23:50.876 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.142.0.42:7180
[2019-02-19 12:23:50.878 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Successfully found server at /10.142.0.42:7180
[2019-02-19 12:23:50.890 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Changing admin Credentials for Cloudera Manager
[2019-02-19 12:23:50.923 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Setting Cloudera Manager License
[2019-02-19 12:23:50.950 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.LicenseClouderaManager - c.c.launchpad.pipeline.AbstractJob: Enabling Enterprise Trial
[2019-02-19 12:23:51.913 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Configuring Cloudera Manager
[2019-02-19 12:23:52.197 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.ConfigureClouderaManager - c.c.l.b.d.ConfigureClouderaManager: Faster Bootstrap is on by default for Cloudera Manager 6.0 and later
[2019-02-19 12:23:52.557 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Deploying Cloudera Manager agent
[2019-02-19 12:23:52.753 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:38:52.753Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=true, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:16:05.847Z, lastChecked=2019-02-19T12:16:05.847Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:23:52.753 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:23:52.755 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:38:52.755Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=true, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:16:05.847Z, lastChecked=2019-02-19T12:16:05.847Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:23:52.755 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:23:52.876 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Using host ID ef94d907-6eb8-4b25-b3fd-79fdc951c1fb for instance with private IP 10.142.0.42.
[2019-02-19 12:23:52.876 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Agent is not already installed or this is a cluster node under auto-TLS, using normal host install.
[2019-02-19 12:23:52.877 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Forming host install arguments for deployment Deployment{name='dep1', hostname='10.142.0.42', port=7180, username='admin', tlsEnabled=false, tlsConfigurationProperties={}, managerInstance=Optional.of(PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=true, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:16:05.847Z, lastChecked=2019-02-19T12:16:05.847Z}, providerInstanceId=Optional.absent()}), createdExternalDatabases=[], repository='Optional.absent()', repositoryKeyUrl='Optional.absent()', repositoryKeyBundleUrl='Optional.absent()', enableEnterpriseTrial=Optional.of(true), unlimitedJce=Optional.absent(), krbAdminUsername='Optional.absent()', javaInstallationStrategy='AUTO', tunnelingRequired=false, cmVersion=Optional.absent()}
[2019-02-19 12:23:52.877 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall - c.c.l.b.c.BootstrapClouderaManagerAgent: Host installation will be performed for host name 10.142.0.42
[2019-02-19 12:23:53.883 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Waiting for Cloudera Manager to deploy agent on 10.142.0.42
[2019-02-19 12:23:53.906 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.launchpad.cm.ApiCommandMonitor: Waiting for commands: [14]
[2019-02-19 12:25:05.744 +0000] INFO [task-thread-2] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-19 12:25:05.745 +0000] INFO [task-thread-2] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-19 12:25:40.527 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.b.c.BootstrapClouderaManagerAgent: Command GlobalHostInstall with ID 14 completed successfully on instance 10.142.0.42. Details: ApiCommand{id=14, name=GlobalHostInstall, startTime=Tue Feb 19 12:23:53 UTC 2019, endTime=Tue Feb 19 12:25:26 UTC 2019, active=false, success=true, resultMessage=Successfully completed installation., serviceRef=null, roleRef=null, hostRef=null, parent=null}
[2019-02-19 12:25:45.900 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Setting up Cloudera Management Services
[2019-02-19 12:25:46.183 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.SetupClouderaManagementServices$CreateMgmtService - c.c.l.b.d.SetupClouderaManagementServices: Auto assigning management service roles
[2019-02-19 12:25:52.753 +0000] INFO [task-thread-2] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2019-02-19 12:25:52.753 +0000] INFO [task-thread-2] - - - - - c.c.l.task.RefreshDeployments: Skipping refresh of deployment env1:dep1 as it is in transition or terminated. (Stage: BOOTSTRAPPING, deploymentIsNull: true)
[2019-02-19 12:25:52.753 +0000] INFO [task-thread-2] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models
[2019-02-19 12:25:55.030 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.ConfigureClouderaManagementServices - c.c.l.b.d.ConfigureClouderaManagementServices: Restarting Cloudera Management Services: ApiCommand{id=24, name=Restart, startTime=Tue Feb 19 12:25:52 UTC 2019, endTime=null, active=true, success=null, resultMessage=null, serviceRef=ApiServiceRef{peerName=null, clusterName=null, serviceName=MGMT}, roleRef=null, hostRef=null, parent=null}
[2019-02-19 12:25:55.035 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Backing up Cloudera Manager Server configuration
[2019-02-19 12:25:55.046 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:40:55.046Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=true, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:16:05.847Z, lastChecked=2019-02-19T12:16:05.847Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:25:55.046 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:25:55.549 +0000] INFO [io-thread-8] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory
[2019-02-19 12:25:55.549 +0000] INFO [io-thread-8] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory
[2019-02-19 12:25:55.749 +0000] INFO [io-thread-8] - - - - - ssh:34.73.121.164: tar: Removing leading `/' from member names
[2019-02-19 12:25:56.236 +0000] INFO [io-thread-1] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: 25a46982)
[2019-02-19 12:25:56.706 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Inspecting capabilities of 10.142.0.42
[2019-02-19 12:25:56.715 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:40:56.715Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.absent(), javaVersion=Optional.absent(), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=true, iptablesEnabled=true, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=false, customScriptPaths={}}), cmHostId=Optional.absent(), cmHostUrl=Optional.absent(), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:16:05.847Z, lastChecked=2019-02-19T12:16:05.847Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:25:56.715 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:25:56.715 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.c.ssh.SshClientPoolFactory: Created new ssh client com.cloudera.launchpad.sshj.SshJClient@77966d95 for endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@672dd79a[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:25:56.965 +0000] INFO [io-thread-5] - - - - - ssh:34.73.121.164: Permissive
[2019-02-19 12:25:56.965 +0000] INFO [io-thread-6] - - - - - ssh:34.73.121.164: director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal
[2019-02-19 12:25:56.984 +0000] INFO [io-thread-10] - - - - - ssh:34.73.121.164: Python 2.7.5
[2019-02-19 12:25:56.992 +0000] INFO [io-thread-7] - - - - - ssh:34.73.121.164: CentOS Linux release 7.6.1810 (Core)
[2019-02-19 12:25:56.992 +0000] INFO [io-thread-7] - - - - - c.c.l.i.OperatingSystemCapabilitiesInspector: Using file /etc/centos-release
[2019-02-19 12:25:57.021 +0000] WARN [reader] - - - - - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 34.73.121.164 was automatically accepted
[2019-02-19 12:25:57.088 +0000] INFO [io-thread-3] - - - - - ssh:34.73.121.164: uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[2019-02-19 12:25:57.312 +0000] INFO [io-thread-6] - - - - - ssh:34.73.121.164: Server: 169.254.169.254
[2019-02-19 12:25:57.312 +0000] INFO [io-thread-6] - - - - - ssh:34.73.121.164: Address: 169.254.169.254#53
[2019-02-19 12:25:57.312 +0000] INFO [io-thread-6] - - - - - ssh:34.73.121.164: Non-authoritative answer:
[2019-02-19 12:25:57.312 +0000] INFO [io-thread-6] - - - - - ssh:34.73.121.164: Name: director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal
[2019-02-19 12:25:57.312 +0000] INFO [io-thread-6] - - - - - ssh:34.73.121.164: Address: 10.142.0.42
[2019-02-19 12:25:57.347 +0000] INFO [io-thread-6] - - - - - c.c.l.i.OperatingSystemCapabilitiesInspector: nslookup output: Server: 169.254.169.254
Address: 169.254.169.254#53
Non-authoritative answer:
Name: director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal
Address: 10.142.0.42
[2019-02-19 12:25:57.845 +0000] INFO [io-thread-8] - - - - - ssh:34.73.121.164: /usr/java/jdk1.8.0_181-cloudera
[2019-02-19 12:25:57.845 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.inspector.OperatingSystemCapabilitiesInspector - c.c.l.inspector.util.JdkInspector: Found JAVA_HOME: /usr/java/jdk1.8.0_181-cloudera
[2019-02-19 12:25:59.207 +0000] INFO [io-thread-1] - - - - - ssh:34.73.121.164: java version "1.8.0_181"
[2019-02-19 12:25:59.207 +0000] INFO [io-thread-1] - - - - - ssh:34.73.121.164: Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
[2019-02-19 12:25:59.207 +0000] INFO [io-thread-1] - - - - - ssh:34.73.121.164: Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
[2019-02-19 12:26:01.511 +0000] INFO [io-thread-1] - - - - - ssh:34.73.121.164: javac 1.8.0_181
[2019-02-19 12:26:02.842 +0000] INFO [task-thread-2] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-19 12:26:02.846 +0000] INFO [task-thread-2] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-19 12:26:05.544 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Running Deployment post create scripts
[2019-02-19 12:26:05.585 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-19 12:26:06.046 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-19 12:26:06.494 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.l.b.c.util.BootstrapClusterUtils: Found Cloudera Manager host ID ef94d907-6eb8-4b25-b3fd-79fdc951c1fb for instance with IP 10.142.0.42.
[2019-02-19 12:26:06.494 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments com.cloudera.launchpad.bootstrap.deployment.RefreshDeployment - c.c.l.b.c.util.BootstrapClusterUtils: Found Cloudera Manager host URL http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb for instance with IP 10.142.0.42.
[2019-02-19 12:26:06.779 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments - - c.c.l.p.s.PipelineRepositoryService: Pipeline '7b5a8dde-fcd5-4623-a164-2469cda306ec': RUNNING -> COMPLETED
[2019-02-19 12:26:06.801 +0000] INFO [p-2469cda306ec-DefaultBootstrapDeploymentJob] d2c2e7b4-9d25-4fed-a7a8-4db9d12a8fbb POST /api/d6.1/environments/env1/deployments - - c.c.l.d.DeploymentRepositoryService: Deployment 'dep1': BOOTSTRAPPING -> READY
[2019-02-19 12:28:31.601 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:29:28.511 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:29:31.546 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:30:01.551 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:30:05.745 +0000] INFO [task-thread-4] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-19 12:30:05.745 +0000] INFO [task-thread-4] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-19 12:30:31.548 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:30:52.757 +0000] INFO [task-thread-4] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2019-02-19 12:30:52.760 +0000] INFO [task-thread-4] - - - - - c.c.l.task.RefreshDeployments: Refreshing deployment env1:dep1.
[2019-02-19 12:30:52.761 +0000] INFO [task-thread-4] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-19 12:30:53.196 +0000] INFO [task-thread-4] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-19 12:30:53.439 +0000] ERROR [task-thread-4] - - - - - c.c.l.b.d.util.DeploymentRefresher: Exception while trying to populate CM version.
[2019-02-19 12:30:53.440 +0000] INFO [task-thread-4] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:45:53.440Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:26:06.268Z, lastChecked=2019-02-19T12:26:06.268Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:30:53.440 +0000] INFO [task-thread-4] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:31:01.540 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:31:02.846 +0000] INFO [task-thread-3] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-19 12:31:02.850 +0000] INFO [task-thread-3] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-19 12:31:03.664 +0000] INFO [io-thread-10] - - - - - ssh:34.73.121.164: https://archive.cloudera.com/cm6/6.1.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.1-853290.el7.x86_64.rpm
[2019-02-19 12:31:03.985 +0000] INFO [task-thread-4] - - - - - c.c.l.b.d.util.DeploymentRefresher: Detected a change in the CM Server Package, setting the CM Repository URL to https://archive.cloudera.com/cm6/6.1.1/redhat7/yum
[2019-02-19 12:31:03.986 +0000] INFO [task-thread-4] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:46:03.986Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:26:06.268Z, lastChecked=2019-02-19T12:26:06.268Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:31:03.986 +0000] INFO [task-thread-4] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:31:04.371 +0000] INFO [task-thread-4] - - - - - c.c.l.task.RefreshDeployments: Backing up Cloudera Manager configuration for Deployment env1:dep1
[2019-02-19 12:31:04.372 +0000] INFO [task-thread-4] - - - - - c.c.l.p.DatabasePipelineService: Starting pipeline 'ab3b3b81-fd30-49ee-98c7-1559783aade3' with root job com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2019-02-19 12:31:04.386 +0000] INFO [task-thread-4] - - - - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline 'ab3b3b81-fd30-49ee-98c7-1559783aade3'
[2019-02-19 12:31:04.400 +0000] INFO [p-1559783aade3-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:46:04.400Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:30:53.424Z, lastChecked=2019-02-19T12:30:53.424Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:31:04.400 +0000] INFO [p-1559783aade3-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:31:05.126 +0000] INFO [io-thread-3] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory
[2019-02-19 12:31:05.126 +0000] INFO [io-thread-3] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory
[2019-02-19 12:31:05.426 +0000] INFO [io-thread-3] - - - - - ssh:34.73.121.164: tar: Removing leading `/' from member names
[2019-02-19 12:31:06.185 +0000] INFO [p-1559783aade3-BackupClouderaManagerConfig] - - - - - c.c.l.p.s.PipelineRepositoryService: Pipeline 'ab3b3b81-fd30-49ee-98c7-1559783aade3': RUNNING -> COMPLETED
[2019-02-19 12:31:06.393 +0000] INFO [task-thread-4] - - - - - c.c.l.p.DatabasePipelineService: Deleting pipeline 'ab3b3b81-fd30-49ee-98c7-1559783aade3'
[2019-02-19 12:31:06.442 +0000] INFO [task-thread-4] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models
[2019-02-19 12:31:06.664 +0000] INFO [io-thread-9] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: 38ccc2e6)
[2019-02-19 12:31:31.540 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:32:01.540 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:32:31.552 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
Contributor
Created 02-19-2019 09:23 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2nd part of logs
[2019-02-19 12:33:01.547 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:33:31.543 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:34:01.540 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:34:31.539 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:35:01.541 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:35:05.745 +0000] INFO [task-thread-8] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-19 12:35:05.746 +0000] INFO [task-thread-8] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-19 12:35:31.544 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:36:01.541 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:36:02.850 +0000] INFO [task-thread-8] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-19 12:36:02.853 +0000] INFO [task-thread-8] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-19 12:36:06.447 +0000] INFO [task-thread-8] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2019-02-19 12:36:06.450 +0000] INFO [task-thread-8] - - - - - c.c.l.task.RefreshDeployments: Refreshing deployment env1:dep1.
[2019-02-19 12:36:07.089 +0000] INFO [task-thread-8] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-19 12:36:07.540 +0000] INFO [task-thread-8] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-19 12:36:07.803 +0000] ERROR [task-thread-8] - - - - - c.c.l.b.d.util.DeploymentRefresher: Exception while trying to populate CM version.
[2019-02-19 12:36:07.803 +0000] INFO [task-thread-8] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:51:07.803Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:30:53.424Z, lastChecked=2019-02-19T12:30:53.424Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:36:07.803 +0000] INFO [task-thread-8] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:36:11.829 +0000] INFO [io-thread-6] - - - - - ssh:34.73.121.164: https://archive.cloudera.com/cm6/6.1.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.1-853290.el7.x86_64.rpm
[2019-02-19 12:36:11.964 +0000] INFO [task-thread-8] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:51:11.964Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:30:53.424Z, lastChecked=2019-02-19T12:30:53.424Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:36:11.965 +0000] INFO [task-thread-8] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:36:12.253 +0000] INFO [task-thread-8] - - - - - c.c.l.task.RefreshDeployments: Backing up Cloudera Manager configuration for Deployment env1:dep1
[2019-02-19 12:36:12.254 +0000] INFO [task-thread-8] - - - - - c.c.l.p.DatabasePipelineService: Starting pipeline '05643996-51b5-4403-9234-d7cc0fef7619' with root job com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2019-02-19 12:36:12.281 +0000] INFO [task-thread-8] - - - - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '05643996-51b5-4403-9234-d7cc0fef7619'
[2019-02-19 12:36:12.293 +0000] INFO [p-d7cc0fef7619-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:51:12.293Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:36:07.795Z, lastChecked=2019-02-19T12:36:07.795Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:36:12.294 +0000] INFO [p-d7cc0fef7619-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:36:12.817 +0000] INFO [io-thread-5] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory
[2019-02-19 12:36:12.817 +0000] INFO [io-thread-5] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory
[2019-02-19 12:36:13.118 +0000] INFO [io-thread-5] - - - - - ssh:34.73.121.164: tar: Removing leading `/' from member names
[2019-02-19 12:36:13.538 +0000] INFO [p-d7cc0fef7619-BackupClouderaManagerConfig] - - - - - c.c.l.p.s.PipelineRepositoryService: Pipeline '05643996-51b5-4403-9234-d7cc0fef7619': RUNNING -> COMPLETED
[2019-02-19 12:36:13.786 +0000] INFO [task-thread-8] - - - - - c.c.l.p.DatabasePipelineService: Deleting pipeline '05643996-51b5-4403-9234-d7cc0fef7619'
[2019-02-19 12:36:13.812 +0000] INFO [task-thread-8] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models
[2019-02-19 12:36:13.887 +0000] INFO [io-thread-1] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: 41a2f0af)
[2019-02-19 12:36:31.548 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:37:01.541 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:37:31.539 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:38:01.555 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:38:31.541 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:39:01.544 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:39:31.540 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:50.433 +0000] INFO [io-thread-4] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:50.434 +0000] INFO [io-thread-5] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:50.435 +0000] INFO [io-thread-1] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:50.435 +0000] INFO [io-thread-2] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:50.435 +0000] INFO [io-thread-10] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:50.435 +0000] INFO [io-thread-7] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:39:50.436 +0000] INFO [io-thread-3] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}]
[2019-02-19 12:40:01.541 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:40:05.746 +0000] INFO [task-thread-5] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-19 12:40:05.747 +0000] INFO [task-thread-5] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-19 12:40:31.540 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:41:01.540 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:41:02.854 +0000] INFO [task-thread-5] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-19 12:41:02.858 +0000] INFO [task-thread-5] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-19 12:41:13.817 +0000] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2019-02-19 12:41:13.820 +0000] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Refreshing deployment env1:dep1.
[2019-02-19 12:41:13.822 +0000] INFO [task-thread-5] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-19 12:41:14.231 +0000] INFO [task-thread-5] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-19 12:41:14.460 +0000] ERROR [task-thread-5] - - - - - c.c.l.b.d.util.DeploymentRefresher: Exception while trying to populate CM version.
[2019-02-19 12:41:14.461 +0000] INFO [task-thread-5] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:56:14.461Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:36:07.795Z, lastChecked=2019-02-19T12:36:07.795Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:41:14.461 +0000] INFO [task-thread-5] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:41:18.715 +0000] INFO [io-thread-9] - - - - - ssh:34.73.121.164: https://archive.cloudera.com/cm6/6.1.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.1-853290.el7.x86_64.rpm
[2019-02-19 12:41:18.913 +0000] INFO [task-thread-5] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:56:18.913Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:36:07.795Z, lastChecked=2019-02-19T12:36:07.795Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:41:18.913 +0000] INFO [task-thread-5] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:41:19.165 +0000] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Backing up Cloudera Manager configuration for Deployment env1:dep1
[2019-02-19 12:41:19.166 +0000] INFO [task-thread-5] - - - - - c.c.l.p.DatabasePipelineService: Starting pipeline '0da666d3-038a-4d50-a41f-c8f4fa633ba2' with root job com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2019-02-19 12:41:19.182 +0000] INFO [task-thread-5] - - - - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '0da666d3-038a-4d50-a41f-c8f4fa633ba2'
[2019-02-19 12:41:19.202 +0000] INFO [p-c8f4fa633ba2-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:56:19.202Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:41:14.452Z, lastChecked=2019-02-19T12:41:14.452Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:41:19.202 +0000] INFO [p-c8f4fa633ba2-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:41:19.626 +0000] INFO [io-thread-8] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory
[2019-02-19 12:41:19.626 +0000] INFO [io-thread-8] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory
[2019-02-19 12:41:19.927 +0000] INFO [io-thread-8] - - - - - ssh:34.73.121.164: tar: Removing leading `/' from member names
[2019-02-19 12:41:20.390 +0000] INFO [p-c8f4fa633ba2-BackupClouderaManagerConfig] - - - - - c.c.l.p.s.PipelineRepositoryService: Pipeline '0da666d3-038a-4d50-a41f-c8f4fa633ba2': RUNNING -> COMPLETED
[2019-02-19 12:41:20.690 +0000] INFO [task-thread-5] - - - - - c.c.l.p.DatabasePipelineService: Deleting pipeline '0da666d3-038a-4d50-a41f-c8f4fa633ba2'
[2019-02-19 12:41:20.727 +0000] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models
[2019-02-19 12:41:21.033 +0000] INFO [io-thread-4] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: e1dd58f8)
[2019-02-19 12:41:31.542 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:42:01.547 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:42:31.541 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:43:01.540 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:45:05.747 +0000] INFO [task-thread-7] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models
[2019-02-19 12:45:05.748 +0000] INFO [task-thread-7] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models
[2019-02-19 12:46:02.858 +0000] INFO [task-thread-7] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2019-02-19 12:46:02.863 +0000] INFO [task-thread-7] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2019-02-19 12:46:20.733 +0000] INFO [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments
[2019-02-19 12:46:20.736 +0000] INFO [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Refreshing deployment env1:dep1.
[2019-02-19 12:46:21.404 +0000] INFO [task-thread-7] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-19 12:46:21.830 +0000] INFO [task-thread-7] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2019-02-19 12:47:22.089 +0000] ERROR [task-thread-7] - - - - - c.c.l.b.d.util.DeploymentRefresher: Exception while trying to populate CM version.
[2019-02-19 12:47:22.090 +0000] INFO [task-thread-7] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T13:02:22.090Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:41:14.452Z, lastChecked=2019-02-19T12:41:14.452Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:47:22.090 +0000] INFO [task-thread-7] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:47:31.605 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:48:36.746 +0000] INFO [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Backing up Cloudera Manager configuration for Deployment env1:dep1
[2019-02-19 12:48:36.747 +0000] INFO [task-thread-7] - - - - - c.c.l.p.DatabasePipelineService: Starting pipeline '979d2579-4444-4072-8793-c93f1762899c' with root job com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2019-02-19 12:48:36.757 +0000] INFO [task-thread-7] - - - - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '979d2579-4444-4072-8793-c93f1762899c'
[2019-02-19 12:48:36.770 +0000] INFO [p-c93f1762899c-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T13:03:36.770Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:46:22.035Z, lastChecked=2019-02-19T12:46:22.035Z}, providerInstanceId=Optional.absent()}
[2019-02-19 12:48:36.771 +0000] INFO [p-c93f1762899c-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds
[2019-02-19 12:49:01.599 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null
[2019-02-19 12:49:06.784 +0000] ERROR [p-c93f1762899c-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.pipeline.ssh.SshJobFailFastWithOutputLogging - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Failed to execute shell command on UnknownHost:-1
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:526)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:507)
va:748)
Caused by: java.lang.RuntimeException: Failed to execute shell command on UnknownHost:-1
at com.cloudera.launchpad.sshj.SshJClient$2.call(SshJClient.java:449)
Caused by: java.util.concurrent.TimeoutException: Timeout expired
... 16 common frames omitted
[2019-02-19 12:49:42.600 +0000] INFO [io-thread-2] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory
[2019-02-19 12:49:42.601 +0000] INFO [io-thread-2] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory
[2019-02-19 12:49:42.901 +0000] INFO [io-thread-2] - - - - - ssh:34.73.121.164: tar: Removing leading `/' from member names
[2019-02-19 12:49:43.361 +0000] INFO [p-c93f1762899c-BackupClouderaManagerConfig] - - - - - c.c.l.p.s.PipelineRepositoryService: Pipeline '979d2579-4444-4072-8793-c93f1762899c': RUNNING -> COMPLETED
[2019-02-19 12:49:43.395 +0000] INFO [task-thread-7] - - - - - c.c.l.p.DatabasePipelineService: Deleting pipeline '979d2579-4444-4072-8793-c93f1762899c'
[2019-02-19 12:49:43.423 +0000] WARN [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Failed to refresh all Deployments. Deployments with errors = [env1:dep1]
[2019-02-19 12:49:43.680 +0000] INFO [io-thread-10] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: 0d9908df)
Announcements
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
