Director cannot create EC2 - Insufficient number of instances available

[#1]

Hi,

 I am observing a very wierd error/bug during cluster provision in aws with configuration scripts. The process starts and in 2-3 minutes it fails with the message that there is zero instances available. 

Cloudera Director 2.4.1 initializing ...
Connecting to http://localhost:7189
Current user roles: [ROLE_READONLY, ROLE_ADMIN]
Configuration file passes all validation checks.
Creating a new environment...
Creating external database servers if configured...
Creating a new Cloudera Manager...
Creating a new CDH cluster...
* Requesting an instance for Cloudera Manager ... done
* Insufficient number of instances available in time 20 MINUTES ...

The cloudera director logs cotains tons of information, but nothing specific about the issue why aws failed to create the instance, just some logs that the cloudera manager EC2 ip was created. After the fail I am able to log in to the EC2, so maybe the Director is not patient enough to wait until the instance come up.

Also that there is no problem with soft limits in EC2 number of instances, no problem with EBS volumes, I double checked that. And sometimes the script fails immediately, sometimes it goes through and creates the cluster (like 1 of 5 succeeds).

Command:

cloudera-director bootstrap-remote cluster.json --lp.update.parallelBatchSize=1 --lp.bootstrap.parallelBatchSize=1 --lp.cloud.instances.allocate.parallelBatchSize=1 

Log

2017-07-27 11:27:06.559 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.560 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.669 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.671 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.672 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.673 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.061 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.062 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.183 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.185 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.187 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.187 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.563 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.564 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.702 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.704 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.706 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.706 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.065 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.065 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.087 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: << Instance i-0b1a6036f16f92113 got IP 10.197.9.108
[2017-07-27 11:27:08.111 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.c.PluggableComputeProvider: Returning 0 pluggable compute instances
[2017-07-27 11:27:08.111 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Waiting for 0 instances to start running
[2017-07-27 11:27:08.111 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.c.PluggableComputeProvider: Waiting for 0 instances to be running
[2017-07-27 11:27:08.112 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.112 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.113 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.114 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.115 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.116 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.119 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.119 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.120 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.123 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.123 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.127 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineService: Scheduled JobInstanceEntity{id=102889, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@0', stackLevel=3, callCount=0, returnValue=PersistentValueEntity{id=327825, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327822,327823,327824,327821, createdAt=Thu Jul 27 11:27:08 EDT 2017, completedAt=null}
[2017-07-27 11:27:08.127 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.128 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.129 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.132 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.132 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.134 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineService: Scheduled JobInstanceEntity{id=102890, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@1', stackLevel=3, callCount=1, returnValue=PersistentValueEntity{id=327827, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327825,327826, createdAt=Thu Jul 27 11:27:08 EDT 2017, completedAt=null}
[2017-07-27 11:27:08.135 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.136 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.136 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=327827, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c231 ...
[2017-07-27 11:27:08.137 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.137 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.138 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.138 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.140 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.140 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.141 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.142 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.144 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.144 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.145 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.p.DatabasePipelineRunner: About to execute JobInstanceEntity{id=102889, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@0', stackLevel=3, callCount=0, returnValue=PersistentValueEntity{id=327825, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327822,327823,327824,327821, createdAt=2017-07-27 11:27:08.126, completedAt=null}
[2017-07-27 11:27:08.181 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.182 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.182 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.183 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.184 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.188 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.188 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances/4 [Environment{name='CLSEC12-POC Environment', provider=InstanceProviderConfig{type='aws'}, credential ...
[2017-07-27 11:27:08.188 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.PipelineJobProfiler: Profiling class com.cloudera.launchpad.bootstrap.AllocateInstances.GetSuccessfulInstancesAndTerminateFailedInstances
[2017-07-27 11:27:08.188 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: All requested instances failed.

[2017-07-27 11:27:08.189 -0400] INFO  [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: Minimum number of instances (1) not available. Terminating available instances (0) as well.
[2017-07-27 11:27:08.225 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.227 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.229 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.229 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.237 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261)
        at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
        at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
        at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3.runUnchecked(<generated>)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
[2017-07-27 11:27:08.242 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error ErrorInfo{code=ErrorCode{name=INSTANCE_ALLOCATION_TIME_OUT, type=SERVICE, retryable=true, keys=[minRequiredCount, timeoutInMinutes]}, properties={minRequiredCount=1, timeoutInMinutes=20}} in job com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261)
        at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
        at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
        at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3.runUnchecked(<generated>)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
[2017-07-27 11:27:08.244 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Pipeline 'a6a67947-fcc1-46e1-ba31-628c7a7c2311' failed
        at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3
        at com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun:0

[#2]

Yes, the whole cluster (CM and the all the CDH nodes) were sucesfully deployed. But not by the Cloudera Director Server (Web UI), because there is not possibility to load the configuration script. So I used the Cloudera Director client command line tool.

The pity thing is, that these two approaches cannot be combined. So if you deploy the cluster by the command line tool, then you loose the ability to "manage" it from the nice UI. On the other hand, deploying HA clusters with lots of custom options is not possible via UI, because there is no option to load your own configuration file.

And the last whish for Cloudera, please if you provide the wizard UI and step by step cluster provision, it would be very nice to integrate with the client tool, to provide the possibility to export the configured deployemnt to a json file. 

It took me a lot of efforts to find out, how some of the specific settings has to be configured in the template.

Regarding troubleshooting, try to look at the Cloudera Director logs. You can increase the verbosity. Basically in the log you will find many concurrent processes writing to the same log, so it needs a little work to extract just the messages related to the provision of Cloudera Manager.

[#3]

Hi Tomas79,

First, thanks for your contributions to this thread as well as your suggestions!

We do steer experienced users toward using configuration files and the bootstrap-remote CLI command vs. the UI, because the UI would get complex if we tried to add a checkbox or field or form for every Director feature into it. The Director server does have a full set of API endpoints that you can use to make updates to clusters that aren't easy or possible to do over the UI, so I recommend taking a look there. If you go to the /api-console URL for Director, there's an interactive facility for learning about the API and trying it out live.

For example, there is an API endpoint for importing a configuration file directly into the server. It's documented here:

https://www.cloudera.com/documentation/director/latest/topics/director_cluster_config.html#concept_l...

We don't have a corresponding configuration file export API endpoint yet, but you are not the first to suggest it, so be assured that it's on our wish list. In the meantime, the API can help you if you're willing to work with that.

Director's single log is tough to navigate. Recent Director versions have added more context to lines in the log which make it feasible to filter relevant lines out. We've got some techniques documented here:

https://www.cloudera.com/documentation/director/latest/topics/director_troubleshoot.html

But I see room for more documentation. Specifically, at least as of 2.4:

• Each line includes a thread ID in square brackets. The ones starting with "p-" are for pipelines, Director's internal workflows, so you can follow one of those among all the other pipelines and other asynchronous tasks within Director.
• The fields following the thread ID are the unique (API) request ID, request method, and request URI that ultimately caused the activity being logged.

You can work with the logback.xml file for the server to change the formatting, and perhaps even route logging to multiple files for easier comprehension (another ask that we've heard).

Again, thanks for your feedback!