Created on 07-27-2017 08:38 AM - edited 09-16-2022 08:45 AM
Hi,
I am observing a very wierd error/bug during cluster provision in aws with configuration scripts. The process starts and in 2-3 minutes it fails with the message that there is zero instances available.
Cloudera Director 2.4.1 initializing ...
Connecting to http://localhost:7189
Current user roles: [ROLE_READONLY, ROLE_ADMIN]
Configuration file passes all validation checks.
Creating a new environment...
Creating external database servers if configured...
Creating a new Cloudera Manager...
Creating a new CDH cluster...
* Requesting an instance for Cloudera Manager ... done
* Insufficient number of instances available in time 20 MINUTES ...
The cloudera director logs cotains tons of information, but nothing specific about the issue why aws failed to create the instance, just some logs that the cloudera manager EC2 ip was created. After the fail I am able to log in to the EC2, so maybe the Director is not patient enough to wait until the instance come up.
Also that there is no problem with soft limits in EC2 number of instances, no problem with EBS volumes, I double checked that. And sometimes the script fails immediately, sometimes it goes through and creates the cluster (like 1 of 5 succeeds).
Command:
cloudera-director bootstrap-remote cluster.json --lp.update.parallelBatchSize=1 --lp.bootstrap.parallelBatchSize=1 --lp.cloud.instances.allocate.parallelBatchSize=1
Log
2017-07-27 11:27:06.559 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:06.560 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:06.669 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:06.671 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:06.672 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:06.673 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.061 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.062 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.183 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.185 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.187 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.187 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.563 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.564 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.702 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.704 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.706 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:07.706 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.065 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.065 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.087 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: << Instance i-0b1a6036f16f92113 got IP 10.197.9.108 [2017-07-27 11:27:08.111 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.c.PluggableComputeProvider: Returning 0 pluggable compute instances [2017-07-27 11:27:08.111 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Waiting for 0 instances to start running [2017-07-27 11:27:08.111 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.c.PluggableComputeProvider: Waiting for 0 instances to be running [2017-07-27 11:27:08.112 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.112 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.113 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.114 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.115 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.116 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.119 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.119 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.120 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.123 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.123 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.127 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineService: Scheduled JobInstanceEntity{id=102889, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@0', stackLevel=3, callCount=0, returnValue=PersistentValueEntity{id=327825, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327822,327823,327824,327821, createdAt=Thu Jul 27 11:27:08 EDT 2017, completedAt=null} [2017-07-27 11:27:08.127 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.128 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.129 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.132 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.132 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.134 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineService: Scheduled JobInstanceEntity{id=102890, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@1', stackLevel=3, callCount=1, returnValue=PersistentValueEntity{id=327827, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327825,327826, createdAt=Thu Jul 27 11:27:08 EDT 2017, completedAt=null} [2017-07-27 11:27:08.135 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.136 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.136 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=327827, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c231 ... [2017-07-27 11:27:08.137 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.137 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.138 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.138 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.140 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.140 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.141 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.142 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.144 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.144 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.145 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.p.DatabasePipelineRunner: About to execute JobInstanceEntity{id=102889, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@0', stackLevel=3, callCount=0, returnValue=PersistentValueEntity{id=327825, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327822,327823,327824,327821, createdAt=2017-07-27 11:27:08.126, completedAt=null} [2017-07-27 11:27:08.181 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.182 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.182 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.183 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.184 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.188 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.188 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances/4 [Environment{name='CLSEC12-POC Environment', provider=InstanceProviderConfig{type='aws'}, credential ... [2017-07-27 11:27:08.188 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.PipelineJobProfiler: Profiling class com.cloudera.launchpad.bootstrap.AllocateInstances.GetSuccessfulInstancesAndTerminateFailedInstances [2017-07-27 11:27:08.188 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: All requested instances failed. [2017-07-27 11:27:08.189 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: Minimum number of instances (1) not available. Terminating available instances (0) as well. [2017-07-27 11:27:08.225 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.227 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.229 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.229 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678 [2017-07-27 11:27:08.237 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307) at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261) at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33) at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) at com.codahale.metrics.Timer.time(Timer.java:101) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) [2017-07-27 11:27:08.242 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error ErrorInfo{code=ErrorCode{name=INSTANCE_ALLOCATION_TIME_OUT, type=SERVICE, retryable=true, keys=[minRequiredCount, timeoutInMinutes]}, properties={minRequiredCount=1, timeoutInMinutes=20}} in job com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307) at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261) at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33) at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) at com.codahale.metrics.Timer.time(Timer.java:101) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) [2017-07-27 11:27:08.244 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Pipeline 'a6a67947-fcc1-46e1-ba31-628c7a7c2311' failed at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3 at com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun:0
Created 07-28-2017 04:03 AM
Created 07-27-2017 10:24 AM
Created 07-27-2017 10:46 AM
Created 07-27-2017 10:55 AM
Created 07-27-2017 12:04 PM
Created 07-27-2017 12:48 PM
Going to the web UI - choosing Clone on the failed cluster (CM is running fine) and executing, failed with the same error. And again, the error message does not correspond to the time, it failed witihin ~1min, no 20,
In AWS logs there is just an entry about creating EC2 and almost immediately after that the termination request.
Created 07-27-2017 04:37 PM
Per application log, the instance was allocated, private ip address assigned, but somehow PluggableComputeProvider complainsed that 0 instance found. It is because allocate and find are two consecutive calls in director, though allocation succeeded, later describe call can still return NOT_FOUND because of eventual consistency of EC2 calls.
Retry will help. To guarantee success, you can set min count to 0, and if the instance is shown not created in director, modify cluster and select retry for the unsatisfied instance.
Created 07-28-2017 04:03 AM
Created 08-04-2017 12:51 AM
Created 09-18-2017 08:34 AM