Thanks for this info (and link) and duly noted on troubleshooting in the future! ... View more

Thanks for the update! ... View more

Hi Bill Thanks a lot for the long explanation. So two options exists (please correct me If I am wrong): 1) Use persistent master nodes and expand for extra temp workloads on demand 2) Alternatively, Have components (Hive, Navigator, etc) save their metadata into S3 or RDS so that all cluster can be teared off and then created from scracth repeately. (That is a better option for cost savings) Many Thanks, Cengiz ... View more

Hi GaryS,   Thanks for following up that you were able to resolve your issue!   For others, to clarify: If you change the username and password for Cloudera Manager (for example, from the default admin/admin), then you do need to update Altus Director with the new credentials. That way, Altus Director can continue to work with Cloudera Manager to do things like add new hosts to a cluster. There is an option in the dropdown for a deployment in Altus Director to update the credentials.   In case there is still a problem in communications, a workaround is to set Cloudera Manager (and Altus Director) back to admin/admin, then do what you need to do, and then switch Cloudera Manager back. There is in fact one scenario in Altus Director where this is necessary, which we're working on fixing.   To add to what Asif said about the ways to add a new cluster node through Director: Besides the UI, you can use the Altus Director server API as well. The UI is just a special client for the API, anyway. Visit http://yourdirectorhost.example:7189/api-console/ for an interactive (Swagger) console to experiment. You can also try using the Java or Python SDKs, available on GitHub.   https://github.com/cloudera/director-sdk ... View more

  2nd part of logs   [2019-02-19 12:33:01.547 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:33:31.543 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:34:01.540 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:34:31.539 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:35:01.541 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:35:05.745 +0000] INFO [task-thread-8] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models [2019-02-19 12:35:05.746 +0000] INFO [task-thread-8] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models [2019-02-19 12:35:31.544 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:36:01.541 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:36:02.850 +0000] INFO [task-thread-8] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting [2019-02-19 12:36:02.853 +0000] INFO [task-thread-8] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting [2019-02-19 12:36:06.447 +0000] INFO [task-thread-8] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments [2019-02-19 12:36:06.450 +0000] INFO [task-thread-8] - - - - - c.c.l.task.RefreshDeployments: Refreshing deployment env1:dep1. [2019-02-19 12:36:07.089 +0000] INFO [task-thread-8] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director' [2019-02-19 12:36:07.540 +0000] INFO [task-thread-8] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director' [2019-02-19 12:36:07.803 +0000] ERROR [task-thread-8] - - - - - c.c.l.b.d.util.DeploymentRefresher: Exception while trying to populate CM version. [2019-02-19 12:36:07.803 +0000] INFO [task-thread-8] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:51:07.803Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:30:53.424Z, lastChecked=2019-02-19T12:30:53.424Z}, providerInstanceId=Optional.absent()} [2019-02-19 12:36:07.803 +0000] INFO [task-thread-8] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds [2019-02-19 12:36:11.829 +0000] INFO [io-thread-6] - - - - - ssh:34.73.121.164: https://archive.cloudera.com/cm6/6.1.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.1-853290.el7.x86_64.rpm [2019-02-19 12:36:11.964 +0000] INFO [task-thread-8] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:51:11.964Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:30:53.424Z, lastChecked=2019-02-19T12:30:53.424Z}, providerInstanceId=Optional.absent()} [2019-02-19 12:36:11.965 +0000] INFO [task-thread-8] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds [2019-02-19 12:36:12.253 +0000] INFO [task-thread-8] - - - - - c.c.l.task.RefreshDeployments: Backing up Cloudera Manager configuration for Deployment env1:dep1 [2019-02-19 12:36:12.254 +0000] INFO [task-thread-8] - - - - - c.c.l.p.DatabasePipelineService: Starting pipeline '05643996-51b5-4403-9234-d7cc0fef7619' with root job com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener [2019-02-19 12:36:12.281 +0000] INFO [task-thread-8] - - - - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '05643996-51b5-4403-9234-d7cc0fef7619' [2019-02-19 12:36:12.293 +0000] INFO [p-d7cc0fef7619-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:51:12.293Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:36:07.795Z, lastChecked=2019-02-19T12:36:07.795Z}, providerInstanceId=Optional.absent()} [2019-02-19 12:36:12.294 +0000] INFO [p-d7cc0fef7619-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds [2019-02-19 12:36:12.817 +0000] INFO [io-thread-5] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory [2019-02-19 12:36:12.817 +0000] INFO [io-thread-5] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory [2019-02-19 12:36:13.118 +0000] INFO [io-thread-5] - - - - - ssh:34.73.121.164: tar: Removing leading `/' from member names [2019-02-19 12:36:13.538 +0000] INFO [p-d7cc0fef7619-BackupClouderaManagerConfig] - - - - - c.c.l.p.s.PipelineRepositoryService: Pipeline '05643996-51b5-4403-9234-d7cc0fef7619': RUNNING -> COMPLETED [2019-02-19 12:36:13.786 +0000] INFO [task-thread-8] - - - - - c.c.l.p.DatabasePipelineService: Deleting pipeline '05643996-51b5-4403-9234-d7cc0fef7619' [2019-02-19 12:36:13.812 +0000] INFO [task-thread-8] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models [2019-02-19 12:36:13.887 +0000] INFO [io-thread-1] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: 41a2f0af) [2019-02-19 12:36:31.548 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:37:01.541 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:37:31.539 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:38:01.555 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:38:31.541 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:39:01.544 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:39:31.540 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:40.431 +0000] INFO [Timer-0] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Removing ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:50.433 +0000] INFO [io-thread-4] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:50.434 +0000] INFO [io-thread-5] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:50.435 +0000] INFO [io-thread-1] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:50.435 +0000] INFO [io-thread-2] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:50.435 +0000] INFO [io-thread-10] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:50.435 +0000] INFO [io-thread-7] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:39:50.436 +0000] INFO [io-thread-3] - - - - - c.c.l.c.ssh.SshClientPoolFactory: Successfully closed ssh client to endpoint(s): com.cloudera.launchpad.common.ssh.SshClientPoolFactory$EndpointsInfo@d62da2e[[10.142.0.42:22, 34.73.121.164:22],SshCredentials{username='gcpcloudera', hasPassword=false, hasPrivateKey=true, hasPassphrase=false, port=22, hostKeyFingerprint=Optional.absent(), bastionHost=Optional.absent()}] [2019-02-19 12:40:01.541 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:40:05.746 +0000] INFO [task-thread-5] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models [2019-02-19 12:40:05.747 +0000] INFO [task-thread-5] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models [2019-02-19 12:40:31.540 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:41:01.540 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:41:02.854 +0000] INFO [task-thread-5] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting [2019-02-19 12:41:02.858 +0000] INFO [task-thread-5] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting [2019-02-19 12:41:13.817 +0000] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments [2019-02-19 12:41:13.820 +0000] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Refreshing deployment env1:dep1. [2019-02-19 12:41:13.822 +0000] INFO [task-thread-5] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director' [2019-02-19 12:41:14.231 +0000] INFO [task-thread-5] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director' [2019-02-19 12:41:14.460 +0000] ERROR [task-thread-5] - - - - - c.c.l.b.d.util.DeploymentRefresher: Exception while trying to populate CM version. [2019-02-19 12:41:14.461 +0000] INFO [task-thread-5] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:56:14.461Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:36:07.795Z, lastChecked=2019-02-19T12:36:07.795Z}, providerInstanceId=Optional.absent()} [2019-02-19 12:41:14.461 +0000] INFO [task-thread-5] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds [2019-02-19 12:41:18.715 +0000] INFO [io-thread-9] - - - - - ssh:34.73.121.164: https://archive.cloudera.com/cm6/6.1.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.1.1-853290.el7.x86_64.rpm [2019-02-19 12:41:18.913 +0000] INFO [task-thread-5] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:56:18.913Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:36:07.795Z, lastChecked=2019-02-19T12:36:07.795Z}, providerInstanceId=Optional.absent()} [2019-02-19 12:41:18.913 +0000] INFO [task-thread-5] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds [2019-02-19 12:41:19.165 +0000] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Backing up Cloudera Manager configuration for Deployment env1:dep1 [2019-02-19 12:41:19.166 +0000] INFO [task-thread-5] - - - - - c.c.l.p.DatabasePipelineService: Starting pipeline '0da666d3-038a-4d50-a41f-c8f4fa633ba2' with root job com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener [2019-02-19 12:41:19.182 +0000] INFO [task-thread-5] - - - - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '0da666d3-038a-4d50-a41f-c8f4fa633ba2' [2019-02-19 12:41:19.202 +0000] INFO [p-c8f4fa633ba2-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T12:56:19.202Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:41:14.452Z, lastChecked=2019-02-19T12:41:14.452Z}, providerInstanceId=Optional.absent()} [2019-02-19 12:41:19.202 +0000] INFO [p-c8f4fa633ba2-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds [2019-02-19 12:41:19.626 +0000] INFO [io-thread-8] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory [2019-02-19 12:41:19.626 +0000] INFO [io-thread-8] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory [2019-02-19 12:41:19.927 +0000] INFO [io-thread-8] - - - - - ssh:34.73.121.164: tar: Removing leading `/' from member names [2019-02-19 12:41:20.390 +0000] INFO [p-c8f4fa633ba2-BackupClouderaManagerConfig] - - - - - c.c.l.p.s.PipelineRepositoryService: Pipeline '0da666d3-038a-4d50-a41f-c8f4fa633ba2': RUNNING -> COMPLETED [2019-02-19 12:41:20.690 +0000] INFO [task-thread-5] - - - - - c.c.l.p.DatabasePipelineService: Deleting pipeline '0da666d3-038a-4d50-a41f-c8f4fa633ba2' [2019-02-19 12:41:20.727 +0000] INFO [task-thread-5] - - - - - c.c.l.task.RefreshDeployments: Finished refreshing all pre-existing Deployment models [2019-02-19 12:41:21.033 +0000] INFO [io-thread-4] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: e1dd58f8) [2019-02-19 12:41:31.542 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:42:01.547 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:42:31.541 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:43:01.540 +0000] INFO [notification-cache-thread-2] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:45:05.747 +0000] INFO [task-thread-7] - - - - - c.c.launchpad.task.RefreshClusters: Refreshing Cluster models [2019-02-19 12:45:05.748 +0000] INFO [task-thread-7] - - - - - c.c.launchpad.task.RefreshClusters: Finished refreshing all pre-existing Cluster models [2019-02-19 12:46:02.858 +0000] INFO [task-thread-7] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting [2019-02-19 12:46:02.863 +0000] INFO [task-thread-7] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting [2019-02-19 12:46:20.733 +0000] INFO [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Refreshing pre-existing Deployments [2019-02-19 12:46:20.736 +0000] INFO [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Refreshing deployment env1:dep1. [2019-02-19 12:46:21.404 +0000] INFO [task-thread-7] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director' [2019-02-19 12:46:21.830 +0000] INFO [task-thread-7] - - - - - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director' [2019-02-19 12:47:22.089 +0000] ERROR [task-thread-7] - - - - - c.c.l.b.d.util.DeploymentRefresher: Exception while trying to populate CM version. [2019-02-19 12:47:22.090 +0000] INFO [task-thread-7] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T13:02:22.090Z to discover SSH service endpoints for instance SimpleInstance{ipAddress='10.142.0.42', hostEndpoints='[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]', properties={launchTime=Tue Feb 19 12:15:42 UTC 2019, imageId=centos-7-v20190213, instanceId=director-d590950c-8949-426a-be11-ded189f7b581, publicIpAddress=34.73.121.164, instanceType=n1-standard-1, privateIpAddress=10.142.0.42}} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:41:14.452Z, lastChecked=2019-02-19T12:41:14.452Z}, providerInstanceId=Optional.absent()} [2019-02-19 12:47:22.090 +0000] INFO [task-thread-7] - - - - - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds [2019-02-19 12:47:31.605 +0000] INFO [notification-cache-thread-3] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:48:36.746 +0000] INFO [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Backing up Cloudera Manager configuration for Deployment env1:dep1 [2019-02-19 12:48:36.747 +0000] INFO [task-thread-7] - - - - - c.c.l.p.DatabasePipelineService: Starting pipeline '979d2579-4444-4072-8793-c93f1762899c' with root job com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener [2019-02-19 12:48:36.757 +0000] INFO [task-thread-7] - - - - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '979d2579-4444-4072-8793-c93f1762899c' [2019-02-19 12:48:36.770 +0000] INFO [p-c93f1762899c-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Waiting until 2019-02-19T13:03:36.770Z to discover SSH service endpoints for instance PluggableComputeInstance{ipAddress=10.142.0.42, delegate=null, hostEndpoints=[HostEndpoint{hostAddressString='10.142.0.42', hostAddress=Optional.of(/10.142.0.42)}, HostEndpoint{hostAddressString='34.73.121.164', hostAddress=Optional.of(/34.73.121.164)}]} Instance{virtualInstance=VirtualInstance{id='d590950c-8949-426a-be11-ded189f7b581', template=InstanceTemplate{name='centos-7-6', type='n1-standard-1', image='https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20190213', rackId='/default', bootstrapScriptsArePresent=true, preTerminateScriptsArePresent=false, config={usePreemptibleInstances=false, dataDiskSizeGb=60, instanceNamePrefix=director, zone=us-east1-b, bootDiskType=Standard, networkName=default, dataDiskType=Standard, dataDiskCount=1, bootDiskSizeGb=60}, tags={}, normalizeInstance=true, normalizationConfig=Optional.absent(), sshUsername=Optional.absent(), sshHostKeyRetrievalType=NONE, groupId=Optional.of(6ff63fff-8be8-4d98-a9d5-90e8cba9f98e)}}, capabilities=Optional.of(Capabilities{operatingSystemType=REDHAT_COMPATIBLE, operatingSystemVersion=REDHAT_COMPATIBLE_7, virtualizationType=UNKNOWN, packageManager=Optional.of(YUM), javaVendor=Optional.of(ORACLE), javaVersion=Optional.of(1.8.0_181), pythonVersion=Optional.of(2.7.5), passwordlessSudoEnabled=true, selinuxEnabled=false, iptablesEnabled=false, dnsConfigured=true, fqdn=Optional.of(director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal), clouderaManagerAgentInstalled=true, customScriptPaths={}}), cmHostId=Optional.of(ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), cmHostUrl=Optional.of(http://director-d590950c-8949-426a-be11-ded189f7b581.us-east1-b.c.cloudera-232012.internal:7180/cmf/hostRedirect/ef94d907-6eb8-4b25-b3fd-79fdc951c1fb), hostKeyFingerprints=[], validationConditions=[], state=InstanceState{status=RUNNING, lastReported=2019-02-19T12:46:22.035Z, lastChecked=2019-02-19T12:46:22.035Z}, providerInstanceId=Optional.absent()} [2019-02-19 12:48:36.771 +0000] INFO [p-c93f1762899c-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.bootstrap.deployment.BackupClouderaManagerConfig - c.c.l.m.e.InstanceServiceEndpoints: Checking every 30000 milliseconds [2019-02-19 12:49:01.599 +0000] INFO [notification-cache-thread-1] - - - - - c.c.l.c.n.DeploymentPingStatusCacheLoader: Cannot ping CM DeploymentKey{environmentName='env1', deploymentName='dep1'} null [2019-02-19 12:49:06.784 +0000] ERROR [p-c93f1762899c-BackupClouderaManagerConfig] - - - com.cloudera.launchpad.pipeline.ssh.SshJobFailFastWithOutputLogging - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed java.util.concurrent.ExecutionException: java.lang.RuntimeException: Failed to execute shell command on UnknownHost:-1 at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:526) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:507) va:748) Caused by: java.lang.RuntimeException: Failed to execute shell command on UnknownHost:-1 at com.cloudera.launchpad.sshj.SshJClient$2.call(SshJClient.java:449) Caused by: java.util.concurrent.TimeoutException: Timeout expired ... 16 common frames omitted [2019-02-19 12:49:42.600 +0000] INFO [io-thread-2] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-agent/agent-cert: No such file or directory [2019-02-19 12:49:42.601 +0000] INFO [io-thread-2] - - - - - ssh:34.73.121.164: ls: cannot access /var/lib/cloudera-scm-server/certmanager: No such file or directory [2019-02-19 12:49:42.901 +0000] INFO [io-thread-2] - - - - - ssh:34.73.121.164: tar: Removing leading `/' from member names [2019-02-19 12:49:43.361 +0000] INFO [p-c93f1762899c-BackupClouderaManagerConfig] - - - - - c.c.l.p.s.PipelineRepositoryService: Pipeline '979d2579-4444-4072-8793-c93f1762899c': RUNNING -> COMPLETED [2019-02-19 12:49:43.395 +0000] INFO [task-thread-7] - - - - - c.c.l.p.DatabasePipelineService: Deleting pipeline '979d2579-4444-4072-8793-c93f1762899c' [2019-02-19 12:49:43.423 +0000] WARN [task-thread-7] - - - - - c.c.l.task.RefreshDeployments: Failed to refresh all Deployments. Deployments with errors = [env1:dep1] [2019-02-19 12:49:43.680 +0000] INFO [io-thread-10] - - - - - c.c.l.p.ssh.SshJobFireAndForget: Succeeded in running script (crc32: 0d9908df) ... View more

To use aes encryption you need to install jce.   Be aware also that encryption types on cloudera side should be inline with kerberos server configuration. ... View more

Hello yarivgraf,   Good news: A community member implemented subnetwork support in the Google plugin, and the work was merged several weeks ago.   https://github.com/cloudera/director-google-plugin/pull/150   The next 6.x release of Cloudera Altus Director will come packaged with a new plugin release that includes this change. In the meantime, you can build the plugin and install it into your existing Altus Director 2.8 or 6.0 installations and it should work.   * For Director 2.8, build and install the plugin from the v1.x branch. * For Director 6.0, build and install the plugin from the v2.0.x branch.   The plugin's README describes the build process and links to some docs on installing the plugin in Altus Director. For the latter, you basically place the plugin JAR into Altus Director's plugins directory, replacing the prior Google plugin JAR. ... View more

Bill,    Thank you for the clarification.      ... View more