Member since
07-01-2015
460
Posts
78
Kudos Received
43
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1346 | 11-26-2019 11:47 PM | |
| 1304 | 11-25-2019 11:44 AM | |
| 9471 | 08-07-2019 12:48 AM | |
| 2175 | 04-17-2019 03:09 AM | |
| 3485 | 02-18-2019 12:23 AM |
11-26-2017
11:49 AM
Exactly. During parcel activation the correct HADOOP_HOME is set to /opt/cloudera/parcels.. etc.
... View more
11-26-2017
11:47 AM
I dont think so. I remember that in SSL it verifies the EXACT name of host, so the CN in the certificate has to be EXACTLY the same as a FQDN of the host.
... View more
09-28-2017
04:06 PM
1 Kudo
How to set DEBUG log level on agent?
... View more
09-18-2017
01:04 PM
Yes, the whole cluster (CM and the all the CDH nodes) were sucesfully deployed. But not by the Cloudera Director Server (Web UI), because there is not possibility to load the configuration script. So I used the Cloudera Director client command line tool. The pity thing is, that these two approaches cannot be combined. So if you deploy the cluster by the command line tool, then you loose the ability to "manage" it from the nice UI. On the other hand, deploying HA clusters with lots of custom options is not possible via UI, because there is no option to load your own configuration file. And the last whish for Cloudera, please if you provide the wizard UI and step by step cluster provision, it would be very nice to integrate with the client tool, to provide the possibility to export the configured deployemnt to a json file. It took me a lot of efforts to find out, how some of the specific settings has to be configured in the template. Regarding troubleshooting, try to look at the Cloudera Director logs. You can increase the verbosity. Basically in the log you will find many concurrent processes writing to the same log, so it needs a little work to extract just the messages related to the provision of Cloudera Manager.
... View more
08-04-2017
12:51 AM
1 Kudo
Using the newer version of Director and using the Oracle JDK I am able to provision the EC2 instances from cloudera-director-client.
... View more
07-28-2017
04:03 AM
1 Kudo
Thanks, but I am setting up everything by script and I want to have a guarantee that if the script with remote-bootstrap is executed, then it will succeed eventually (During the process if something fails, I would assume that Cloudera Director will do a retry and after some attempts it should drop the whole bootstrap) From the log it is clear, that there is no second attempt during the bootstrap, and from the times, it simply fails (or claims to be failed) too early. But I am testing the same process on different Java and with newer Director (2.5). I will let you know how it is goin
... View more
07-27-2017
12:48 PM
Going to the web UI - choosing Clone on the failed cluster (CM is running fine) and executing, failed with the same error. And again, the error message does not correspond to the time, it failed witihin ~1min, no 20, In AWS logs there is just an entry about creating EC2 and almost immediately after that the termination request.
... View more
07-27-2017
12:04 PM
After reconfiguring all the instance types to m4.2xlarge the CM was created sucefully but during cluster EC2 intances it failed and Bootstrapping failed. But I checked in AWS, the config contains 10+1(CM) instances, and in AWS there is 11 instances, 2 terminated. So the EC2 were created and running, but Director did not waited enough time. So now I am having CM running, but the Cluster in "Boostrap failed". During the provision I checked the Director UI and when the progress bar was increasing I could see in the hover that 10 servers are in Starting. Then two of them changed to Insuf num with 20MIN and BOOTSTRAP failed. In AWS there is 1 EC2 running (CM) and 10 EC2 for Cluster from which 2 are terminated.
... View more
07-27-2017
10:46 AM
General purpose m4 4xlarge. The region is eu-west-1 and I didnt used placement group. The problem is that the error message is raise after 2 or 3 minutes after command submit. Tge director is NOT waiting 20min before raising an excrption
... View more
07-27-2017
08:38 AM
Hi, I am observing a very wierd error/bug during cluster provision in aws with configuration scripts. The process starts and in 2-3 minutes it fails with the message that there is zero instances available. Cloudera Director 2.4.1 initializing ... Connecting to http://localhost:7189 Current user roles: [ROLE_READONLY, ROLE_ADMIN] Configuration file passes all validation checks. Creating a new environment... Creating external database servers if configured... Creating a new Cloudera Manager... Creating a new CDH cluster... * Requesting an instance for Cloudera Manager ... done * Insufficient number of instances available in time 20 MINUTES ... The cloudera director logs cotains tons of information, but nothing specific about the issue why aws failed to create the instance, just some logs that the cloudera manager EC2 ip was created. After the fail I am able to log in to the EC2, so maybe the Director is not patient enough to wait until the instance come up. Also that there is no problem with soft limits in EC2 number of instances, no problem with EBS volumes, I double checked that. And sometimes the script fails immediately, sometimes it goes through and creates the cluster (like 1 of 5 succeeds). Command: cloudera-director bootstrap-remote cluster.json --lp.update.parallelBatchSize=1 --lp.bootstrap.parallelBatchSize=1 --lp.cloud.instances.allocate.parallelBatchSize=1 Log 2017-07-27 11:27:06.559 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.560 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.669 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.671 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.672 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:06.673 -0400] DEBUG [qtp1260487756-17] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.061 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.062 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.183 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.185 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.187 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.187 -0400] DEBUG [qtp1260487756-18] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.563 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.564 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.702 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.704 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.706 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:07.706 -0400] DEBUG [qtp1260487756-19] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.065 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.065 -0400] DEBUG [p-e1a91da67677-DefaultBootstrapClusterJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.087 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: << Instance i-0b1a6036f16f92113 got IP 10.197.9.108
[2017-07-27 11:27:08.111 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.c.PluggableComputeProvider: Returning 0 pluggable compute instances
[2017-07-27 11:27:08.111 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Waiting for 0 instances to start running
[2017-07-27 11:27:08.111 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.c.PluggableComputeProvider: Waiting for 0 instances to be running
[2017-07-27 11:27:08.112 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.112 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.113 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.114 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.115 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.116 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.119 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.119 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.120 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.121 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.123 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.123 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.127 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineService: Scheduled JobInstanceEntity{id=102889, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@0', stackLevel=3, callCount=0, returnValue=PersistentValueEntity{id=327825, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327822,327823,327824,327821, createdAt=Thu Jul 27 11:27:08 EDT 2017, completedAt=null}
[2017-07-27 11:27:08.127 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.128 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.129 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.132 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.132 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.134 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineService: Scheduled JobInstanceEntity{id=102890, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@1', stackLevel=3, callCount=1, returnValue=PersistentValueEntity{id=327827, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327825,327826, createdAt=Thu Jul 27 11:27:08 EDT 2017, completedAt=null}
[2017-07-27 11:27:08.135 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.136 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.136 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=327827, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c231 ...
[2017-07-27 11:27:08.137 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.137 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.138 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.138 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.140 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.140 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.141 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.142 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.144 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.144 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.145 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.p.DatabasePipelineRunner: About to execute JobInstanceEntity{id=102889, pipeline=PipelineEntity{id=15219, handle='a6a67947-fcc1-46e1-ba31-628c7a7c2311', parent='/CLSEC12-POC Environment/CLSEC12-POC Deployment', statusDescription='Requesting an instance for Cloudera Manager', returnValue=PersistentValueEntity{id=327690, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, stage=RUNNING, stageListenerClass='com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener', createdAt=2017-07-27 11:26:46.165, updatedAt=2017-07-27 11:26:46.563}, jobClass='com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances', callStack='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob@1/com.cloudera.launchpad.bootstrap.AllocateInstances@0/com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun@0', stackLevel=3, callCount=0, returnValue=PersistentValueEntity{id=327825, pipeline=a6a67947-fcc1-46e1-ba31-628c7a7c2311, content=null, listener=null}, arguments=327822,327823,327824,327821, createdAt=2017-07-27 11:27:08.126, completedAt=null}
[2017-07-27 11:27:08.181 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.182 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.182 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.183 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.184 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.185 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.186 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.187 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.188 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.188 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances/4 [Environment{name='CLSEC12-POC Environment', provider=InstanceProviderConfig{type='aws'}, credential ...
[2017-07-27 11:27:08.188 -0400] DEBUG [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.PipelineJobProfiler: Profiling class com.cloudera.launchpad.bootstrap.AllocateInstances.GetSuccessfulInstancesAndTerminateFailedInstances
[2017-07-27 11:27:08.188 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: All requested instances failed.
[2017-07-27 11:27:08.189 -0400] INFO [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: Minimum number of instances (1) not available. Terminating available instances (0) as well.
[2017-07-27 11:27:08.225 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.DeploymentTemplateUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.227 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.database.types.JsonUserType: Decrypting class com.cloudera.launchpad.database.types.EnvironmentUserType using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.229 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.229 -0400] DEBUG [qtp1260487756-14] GET /api/v8/environments/CLSEC12-POC%20Environment/deployments/CLSEC12-POC%20Deployment/status - - c.c.l.c.j.ByteArrayToBlobEncryptionConverter: Decrypting attribute using cipher scheme com.cloudera.launchpad.common.security.TripleDESCipher@2c0f7678
[2017-07-27 11:27:08.237 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307)
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261)
at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
at com.codahale.metrics.Timer.time(Timer.java:101)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
[2017-07-27 11:27:08.242 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error ErrorInfo{code=ErrorCode{name=INSTANCE_ALLOCATION_TIME_OUT, type=SERVICE, retryable=true, keys=[minRequiredCount, timeoutInMinutes]}, properties={minRequiredCount=1, timeoutInMinutes=20}} in job com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307)
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261)
at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
at com.codahale.metrics.Timer.time(Timer.java:101)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
[2017-07-27 11:27:08.244 -0400] ERROR [p-628c7a7c2311-DefaultBootstrapDeploymentJob] POST /api/v8/environments/CLSEC12-POC%20Environment/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Pipeline 'a6a67947-fcc1-46e1-ba31-628c7a7c2311' failed
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$c97dbec3
at com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun:0
... View more
Labels: