Director cannot create EC2 - Insufficient number of instances available

[#1]

Hi,

 I am observing a very wierd error/bug during cluster provision in aws with configuration scripts. The process starts and in 2-3 minutes it fails with the message that there is zero instances available. 

Cloudera Director 2.4.1 initializing ...
Connecting to http://localhost:7189
Current user roles: [ROLE_READONLY, ROLE_ADMIN]
Configuration file passes all validation checks.
Creating a new environment...
Creating external database servers if configured...
Creating a new Cloudera Manager...
Creating a new CDH cluster...
* Requesting an instance for Cloudera Manager ... done
* Insufficient number of instances available in time 20 MINUTES ...

The cloudera director logs cotains tons of information, but nothing specific about the issue why aws failed to create the instance, just some logs that the cloudera manager EC2 ip was created. After the fail I am able to log in to the EC2, so maybe the Director is not patient enough to wait until the instance come up.

Also that there is no problem with soft limits in EC2 number of instances, no problem with EBS volumes, I double checked that. And sometimes the script fails immediately, sometimes it goes through and creates the cluster (like 1 of 5 succeeds).

Command:

cloudera-director bootstrap-remote cluster.json --lp.update.parallelBatchSize=1 --lp.bootstrap.parallelBatchSize=1 --lp.cloud.instances.allocate.parallelBatchSize=1 

Log

2017-07-27 11:27:06.559 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.560 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.669 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.671 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.672 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.673 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.061 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.062 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.183 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.185 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.187 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.187 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.563 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.564 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.702 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.704 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.706 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.706 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.065 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.065 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.087 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: << Instance i-0b1a6036f16f92113 got IP 10.197.9.108
[2017-07-27 11:27:08.111 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.c.PluggableComputeProvider: Returning 0 pluggable compute instances
[2017-07-27 11:27:08.111 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Waiting for 0 instances to start running
[2017-07-27 11:27:08.111 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.c.PluggableComputeProvider: Waiting for 0 instances to be running
[2017-07-27 11:27:08.112 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.112 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.113 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.114 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.115 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.116 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.119 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.119 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.120 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.123 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.123 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.127 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineService: Scheduled JobInstanceEntity{id=102889, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@0', stackLevel=3, callCount=0, returnValue=PersistentValueEntity{id=327825, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327822,327823,327824,327821, createdAt=Thu Jul 27 11:27:08 EDT 2017, completedAt=null}
[2017-07-27 11:27:08.127 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.128 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.129 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.132 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.132 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.134 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineService: Scheduled JobInstanceEntity{id=102890, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@1', stackLevel=3, callCount=1, returnValue=PersistentValueEntity{id=327827, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327825,327826, createdAt=Thu Jul 27 11:27:08 EDT 2017, completedAt=null}
[2017-07-27 11:27:08.135 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.136 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.136 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=327827, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c231 ...
[2017-07-27 11:27:08.137 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.137 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.138 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.138 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.140 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.140 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.141 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.142 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.144 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.144 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.145 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.p.DatabasePipelineRunner: About to execute JobInstanceEntity{id=102889, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@0', stackLevel=3, callCount=0, returnValue=PersistentValueEntity{id=327825, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327822,327823,327824,327821, createdAt=2017-07-27 11:27:08.126, completedAt=null}
[2017-07-27 11:27:08.181 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.182 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.182 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.183 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.184 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.188 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.188 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances/4 [Environment{name='CLSEC12-POC Environment', provider=InstanceProviderConfig{type='aws'}, credential ...
[2017-07-27 11:27:08.188 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.PipelineJobProfiler: Profiling class com.cloudera.launchpad.bootstrap.AllocateInstances.GetSuccessfulInstancesAndTerminateFailedInstances
[2017-07-27 11:27:08.188 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: All requested instances failed.

[2017-07-27 11:27:08.189 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: Minimum number of instances (1) not available. Terminating available instances (0) as well.
[2017-07-27 11:27:08.225 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.227 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.229 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.229 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.237 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261)
        at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
        at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
        at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3.runUnchecked(<generated>)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
[2017-07-27 11:27:08.242 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error ErrorInfo{code=ErrorCode{name=INSTANCE_ALLOCATION_TIME_OUT, type=SERVICE, retryable=true, keys=[minRequiredCount, timeoutInMinutes]}, properties={minRequiredCount=1, timeoutInMinutes=20}} in job com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261)
        at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
        at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
        at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3.runUnchecked(<generated>)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
[2017-07-27 11:27:08.244 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Pipeline 'a6a67947-fcc1-46e1-ba31-628c7a7c2311' failed
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3
        at com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun:0

[#2]

What instance type are you using? Are they on-demand or reserved? What region and zone is this in? Did you use a Placement group?

It could be that you are requesting an instance size, like the d2.8xl, and there just isn't enough room in the region/zone to allocate them in time, 20 mins. It could also be that AWS black magik isn't fast enough in moving instances around if you used a Placement group.

[#3]

General purpose m4 4xlarge. The region is eu-west-1 and I didnt used placement group. The problem is that the error message is raise after 2 or 3 minutes after command submit.
Tge director is NOT waiting 20min before raising an excrption

[#4]

It is possible that AWS is throwing an error that leads Director to state that it can't get the instance in time. The minimum required is only 1. You could test quickly by launching the same size in the same region to see if there is an issue on the AWS side.

[#5]

After reconfiguring all the instance types to m4.2xlarge the CM was created sucefully but during cluster EC2 intances it failed and Bootstrapping failed.

But I checked in AWS, the config contains 10+1(CM) instances, and in AWS there is 11 instances, 2 terminated. So the EC2 were created and running, but Director did not waited enough time. So now I am having CM running, but the Cluster in "Boostrap failed".

During the provision I checked the Director UI and when the progress bar was increasing I could see in the hover that 10 servers are in Starting. Then two of them changed to Insuf num with 20MIN and BOOTSTRAP failed. In AWS there is 1 EC2 running (CM) and 10 EC2 for Cluster from which 2 are terminated.

[#6]

Going to the web UI - choosing Clone on the failed cluster (CM is running fine) and executing, failed with the same error. And again, the error message does not correspond to the time, it failed witihin ~1min, no 20,

In AWS logs there is just an entry about creating EC2 and almost immediately after that the termination request.

[#7]

Per application log, the instance was allocated, private ip address assigned, but somehow PluggableComputeProvider complainsed that 0 instance found. It is because allocate and find are two consecutive calls in director, though allocation succeeded, later describe call can still return NOT_FOUND because of eventual consistency of EC2 calls.

Retry will help. To guarantee success, you can set min count to 0, and if the instance is shown not created in director, modify cluster and select retry for the unsatisfied instance.

[#8 · Accepted solution]

Thanks, but I am setting up everything by script and I want to have a guarantee that if the script with remote-bootstrap is executed, then it will succeed eventually (During the process if something fails, I would assume that Cloudera Director will do a retry and after some attempts it should drop the whole bootstrap)

From the log it is clear, that there is no second attempt during the bootstrap, and from the times, it simply fails (or claims to be failed) too early.

But I am testing the same process on different Java and with newer Director (2.5). I will let you know how it is goin

[#9]

Using the newer version of Director and using the Oracle JDK I am able to provision the EC2 instances from cloudera-director-client.

[#10]

Hi Tomas,

Did the bootstarp for the Cloudera Manager Work for you? I can´t get past that step. Any tips are more than welcome.

best