Director: Cloudera Manager - Bootstrap failed on AWS China region

[#1]

Hi,

I've been struggling with installing Director in China region and finally managed to do it, but now I'm stuck when creating a Manager cluster.

Cloudera Director fails when bootstrapping Manager. The ec2 instances are created but do not start completely, only error message I get is "Server.InternalError: Internal error on launch".

/var/log/cloudera-director-server/application.log:

[2017-11-08 06:39:20.659 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating deployment template: testmanager
[2017-11-08 06:39:20.659 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating parcel repository URLs
[2017-11-08 06:39:20.659 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating CSD URLs
[2017-11-08 06:39:20.659 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating external databases and templates
[2017-11-08 06:39:20.659 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.m.r.v.UsageCollectionHeartbeatDeploymentTemplateValidator: Validating deployment template: testmanager
[2017-11-08 06:39:20.659 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.p.c.PluggableComputeDeploymentTemplateValidator: Validating Cloudera Manager virtual instance template
[2017-11-08 06:39:20.659 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.p.c.PluggableComputeInstanceTemplateValidator: Validating instance template for compute provider: aws
[2017-11-08 06:39:20.797 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.director.aws.ec2.EC2Provider: Found EC2 key name heat-dev for fingerprint
[2017-11-08 06:39:20.797 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing AMI 'ami-3ce23651'
[2017-11-08 06:39:20.822 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing subnet 'subnet-f0b55994'
[2017-11-08 06:39:20.839 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing security group 'sg-4ec4aa2a'
[2017-11-08 06:39:20.860 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing key pair
[2017-11-08 06:39:20.928 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.p.DatabasePipelineService: Starting pipeline '696cc743-68c1-44fe-ac0c-f3c6eeaeb024' with root job com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob and listener com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener
[2017-11-08 06:39:20.933 +0000] INFO  [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '696cc743-68c1-44fe-ac0c-f3c6eeaeb024'
[2017-11-08 06:39:21.039 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.d.DeploymentRepositoryService: Deployment 'testmanager': BOOTSTRAPPING -> BOOTSTRAPPING
[2017-11-08 06:39:21.046 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob - c.c.l.pipeline.util.PipelineRunner: >> DefaultBootstrapDeploymentJob/2 [CreateDeploymentContext{environment=Environment{name='Heat', provider=InstanceProviderConfig{type=' ...
[2017-11-08 06:39:21.073 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=721, pipeline=696cc743-68c1-44fe-ac0c-f3c6eeaeb024,  ...
[2017-11-08 06:39:21.079 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Requesting an instance for Cloudera Manager]
[2017-11-08 06:39:21.080 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Requesting an instance for Cloudera Manager
[2017-11-08 06:39:21.080 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-11-08 06:39:21.089 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances/2 [VirtualInstanceGroup{name='CM', virtualInstances=[VirtualInstance{id='63391619-bfcb-4b6f-9a87-f4af9 ...
[2017-11-08 06:39:21.102 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=730, pipeline=696cc743-68c1-44fe-ac0c-f3c6eeaeb024,  ...
[2017-11-08 06:39:21.110 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$AllocateAndWaitForInstancesToRun/2 [VirtualInstanceGroup{name='CM', virtualInstances=[VirtualInstance{id='63391619-bfcb-4b6f-9a87-f4af9 ...
[2017-11-08 06:39:21.110 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Allocating 1 instances (min count 1) in group CM
[2017-11-08 06:39:21.191 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Found EC2 key name heat-dev for fingerprint
[2017-11-08 06:39:21.191 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Requesting 1 instances for com.cloudera.director.aws.ec2.EC2InstanceTemplate@7d85aff0
[2017-11-08 06:39:21.225 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Building 1 instance requests
[2017-11-08 06:39:21.225 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Network interface specification: {DeviceIndex: 0,SubnetId: subnet-f0b55994,Groups: [sg-4ec4aa2a],DeleteOnTermination: true,PrivateIpAddresses: [],AssociatePublicIpAddress: true,Ipv6Addresses: [],}
[2017-11-08 06:39:21.252 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Original image block device mappings: [{DeviceName: /dev/sda1,Ebs: {SnapshotId: snap-c8cfeefb,VolumeSize: 10,DeleteOnTermination: true,VolumeType: gp2,Encrypted: false},}]
[2017-11-08 06:39:21.252 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Block device mappings: [{DeviceName: /dev/sda1,Ebs: {SnapshotId: snap-c8cfeefb,VolumeSize: 50,DeleteOnTermination: true,VolumeType: gp2,},}]
[2017-11-08 06:39:21.252 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Instance request type: m4.xlarge, image: ami-3ce23651
[2017-11-08 06:39:21.252 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Submitted 1 run instance requests.
[2017-11-08 06:39:22.433 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: << Reservation r-0ac36602c7afb02f6 with Instance{id=i-0f418699fdc2412f8 privateIp=172.31.21.217} 
[2017-11-08 06:39:22.697 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Retryer: resource not found, might be a transient error
com.cloudera.director.aws.shaded.com.amazonaws.AmazonServiceException: InvalidInstanceID.NotFound (Service: null; Status Code: 0; Error Code: InvalidInstanceID.NotFound; Request ID: null)
	at com.cloudera.director.aws.ec2.EC2Provider$10.call(EC2Provider.java:1581)
	at com.cloudera.director.aws.ec2.EC2Provider$10.call(EC2Provider.java:1)
	at com.cloudera.director.aws.shaded.com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
	at com.cloudera.director.aws.shaded.com.github.rholder.retry.Retryer.call(Retryer.java:160)
	at com.cloudera.director.aws.ec2.EC2Retryer.retryUntil(EC2Retryer.java:94)
	at com.cloudera.director.aws.ec2.EC2Retryer.retryUntil(EC2Retryer.java:81)
	at com.cloudera.director.aws.ec2.EC2Retryer.retryUntil(EC2Retryer.java:71)
	at com.cloudera.director.aws.ec2.EC2Provider.waitUntilInstanceHasStarted(EC2Provider.java:1588)
	at com.cloudera.director.aws.ec2.EC2Provider.allocateOnDemandInstances(EC2Provider.java:1405)
	at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:725)
	at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:1)
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:614)
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:545)
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:520)
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:331)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:220)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:196)
	at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
	at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
	at sun.reflect.GeneratedMethodAccessor228.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$604aec91.runUnchecked(<generated>)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
	at com.github.rholder.retry.Retryer.call(Retryer.java:160)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:199)
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:139)
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2017-11-08 06:39:27.808 +0000] ERROR [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Instance i-0f418699fdc2412f8 has unexpectedly terminated, reason Server.InternalError: Internal error on launch
[2017-11-08 06:39:27.808 +0000] INFO  [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: << Instance i-0f418699fdc2412f8 did not start.
[2017-11-08 06:39:27.808 +0000] ERROR [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Trying to allocate (1) instances, (0) instances allocated, below minimum count (1): (0) instances failed for instance limit exceeded, (0) instances failed for request limit exceeded, (1) instances failed when flipping from Pending to Running, including volume limit exceeded.
[2017-11-08 06:39:27.808 +0000] ERROR [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Unsuccessful allocation of on demand instances. Terminating instances.

I've tried different ami images etc. to no avail, any help is appreciated!

[#2]

I was using Director 2.6, downgrading to 2.4 did not have the same problem.

Next problem though: Seems to timeout when downloading yum packages, too slow connection from China.

Does anyone know if using a proxy server works when installing using Cloudera Manager?

[#3]

David,

Cloudera Director 2.5 changed the way it tags instances to include the tags on the runInstances request. This feature has not been added to the China region. I'm afraid that you will have to stay on 2.4.x until Amazon updates that region to support this features.

A proxy server will work.

Which yum packages are timing out? Here are other alternatives.

You could create an ami with the packages pre-installed.

https://github.com/cloudera/director-scripts/tree/master/faster-bootstrap

You could host the parcels and packages. If you do this, then you will need to configure the custom repos when you create a Deployment or Cluster.

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_create_local_package_repo.html

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_create_local_parcel_repo.html

Good Luck!

David

[#4]

Thanks!

I've now managed to setup a local package repo and got a Manager up and running through Director.

Next problem is when adding a cluster, it fails with:

Bootstrap failed

Insufficient number of instances available in time 20 MINUTES

[2017-11-10 00:56:39.555 -0500] INFO  [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.director.aws.ec2.EC2Provider: Found EC2 key name heat-dev for fingerprint
[2017-11-10 00:56:39.560 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10081, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:39.568 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances/4 [Environment{name='Heat', provider=InstanceProviderConfig{type='aws'}, credentials=SshCredentials{us ...
[2017-11-10 00:56:39.568 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: All requested instances are available
[2017-11-10 00:56:39.568 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: Sufficient number of instances available (1/1)
[2017-11-10 00:56:39.584 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10082, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:39.594 -0500] INFO  [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.director.aws.ec2.EC2Provider: >> Terminating [i-01e0d4a7f40831412, i-00dde88bcea48c247]
[2017-11-10 00:56:39.605 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$RefreshInstancesMetadata/2 [[PluggableComputeInstance{ipAddress=172.31.29.211, delegate=null, hostEndpoints=[HostEndpoint{hostA ...
[2017-11-10 00:56:39.715 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata - c.c.director.aws.ec2.EC2Provider: Found EC2 key name heat-dev for fingerprint
[2017-11-10 00:56:39.793 -0500] INFO  [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.director.aws.ec2.EC2Provider: << Result {TerminatingInstances: [{InstanceId: i-00dde88bcea48c247,CurrentState: {Code: 32,Name: shutting-down},PreviousState: {Code: 16,Name: running}}, {InstanceId: i-01e0d4a7f40831412,CurrentState: {Code: 32,Name: shutting-down},PreviousState: {Code: 16,Name: running}}]}
[2017-11-10 00:56:39.794 -0500] ERROR [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES
	at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261)
	at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
	at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
	at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$7ec0d65c.runUnchecked(<generated>)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
	at com.github.rholder.retry.Retryer.call(Retryer.java:160)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2017-11-10 00:56:39.794 -0500] ERROR [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error ErrorInfo{code=ErrorCode{name=INSTANCE_ALLOCATION_TIME_OUT, type=SERVICE, retryable=true, keys=[minRequiredCount, timeoutInMinutes]}, properties={minRequiredCount=3, timeoutInMinutes=20}} in job com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES
	at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261)
	at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
	at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
	at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$7ec0d65c.runUnchecked(<generated>)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
	at com.github.rholder.retry.Retryer.call(Retryer.java:160)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2017-11-10 00:56:39.794 -0500] ERROR [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Pipeline '856656a2-1693-4a66-b7ca-3a8e2a2de99b/child-00000-8b4b0060-a447-4b94-b2a0-ee24d88a4e8f' failed
	at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$7ec0d65c
	at com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun:0
	at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach:1

[2017-11-10 00:56:39.797 -0500] INFO  [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.s.PipelineRepositoryService: Pipeline '856656a2-1693-4a66-b7ca-3a8e2a2de99b/child-00000-8b4b0060-a447-4b94-b2a0-ee24d88a4e8f': RUNNING -> ERROR
[2017-11-10 00:56:39.874 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata - c.c.director.aws.ec2.EC2Provider: >> Fetching page 0
[2017-11-10 00:56:39.908 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata - c.c.director.aws.ec2.EC2Provider: << Result: {InstanceStatuses: [{InstanceId: i-05b4b9efee75811c6,AvailabilityZone: cn-north-1a,Events: [],InstanceState: {Code: 16,Name: running},SystemStatus: {Status: initializing,Details: [{Name: reachability,Status: initializing,}]},InstanceStatus: {Status: initializing,Details: [{Name: reachability,Status: initializing,}]}}],}
[2017-11-10 00:56:39.919 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10083, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:39.948 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForInstancePorts - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$WaitForInstancePorts/3 [[PluggableComputeInstance{ipAddress=172.31.29.211, delegate=null, hostEndpoints=[HostEndpoint{hostA ...
[2017-11-10 00:56:39.950 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForInstancePorts - c.c.l.bootstrap.AllocateInstances: Waiting for SSH port of instances to become available
[2017-11-10 00:56:39.962 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForInstancePorts - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10093, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:39.971 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches - c.c.l.pipeline.util.PipelineRunner: >> ParallelForEachInBatches/6 [20, class com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime, [PluggableComputeIn ...
[2017-11-10 00:56:39.977 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches - c.c.l.p.u.ParallelForEachInBatches: Generating batch for job class com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime of size 1
[2017-11-10 00:56:39.979 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10104, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:39.987 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach - c.c.l.pipeline.util.PipelineRunner: >> UnboundedParallelForEach/5 [class com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime, [PluggableComputeInstan ...
[2017-11-10 00:56:39.988 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach - c.c.l.p.DatabasePipelineService: Starting pipeline '856656a2-1693-4a66-b7ca-3a8e2a2de99b/child-00000-8d962b49-2ac6-4f05-a671-8523472863d1/child-00000-c36d6d33-e2d0-448d-a4e1-34ed3aaf0cef' with root job com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2017-11-10 00:56:39.999 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10114, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:40.003 -0500] INFO  [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForAllPipelines/1 [[856656a2-1693-4a66-b7ca-3a8e2a2de99b/child-00000-8d962b49-2ac6-4f05-a671-8523472863d1/child-00000- ...
[2017-11-10 00:56:40.005 -0500] INFO  [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime - c.c.l.pipeline.util.PipelineRunner: >> WaitForSshAccessOnInstanceUntilTime/4 [PluggableComputeInstance{ipAddress=172.31.29.211, delegate=null, hostEndpoints=[HostEndpoint{hostAd ...
[2017-11-10 00:56:40.017 -0500] INFO  [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10120, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:40.023 -0500] INFO  [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.pipeline.util.PipelineRunner: >> WaitForServersUntilTime/3 [[172.31.29.211:22, ip-172-31-29-211.cn-north-1.compute.internal:22, 52.80.58.19:22, ec2-52-80-58-19 ...
[2017-11-10 00:56:40.023 -0500] INFO  [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Waiting for 1199927 milliseconds for an accessible port on endpoints [172.31.29.211:22, ip-172-31-29-211.cn-north-1.compute.internal:22, 52.80.58.19:22, ec2-52-80-58-19.cn-north-1.compute.amazonaws.com.cn:22]
[2017-11-10 00:56:40.023 -0500] INFO  [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Checking every 10000 milliseconds
[2017-11-10 00:56:40.025 -0500] INFO  [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /172.31.29.211:22
[2017-11-10 00:56:40.526 -0500] INFO  [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to ip-172-31-29-211.cn-north-1.compute.internal/172.31.29.211:22
[2017-11-10 00:56:41.027 -0500] INFO  [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /52.80.58.19:22
[2017-11-10 00:56:41.528 -0500] INFO  [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to ec2-52-80-58-19.cn-north-1.compute.amazonaws.com.cn/52.80.58.19:22
[2017-11-10 00:56:43.701 -0500] INFO  [p-b365a6e939c4-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: << Instance i-0094aa2d6b87403f0 got IP 172.31.21.72
[2017-11-10 00:56:43.727 -0500] INFO  [p-b365a6e939c4-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Waiting for 0 instances to start running

Its strange because I'm getting the above error just after like 1 minute, while the 5 ec2 instances are being initialized. They are actually eventually starting up ok as I can see in the ec2 console, but Director says failed!

I can't seem to find the cause in the logfile? Director seems to both see the instances and think they are not finished and fails too early...

/David

[#5]

I have setup a local package and parcel repo but cluster bootstrap failing with:

[2017-11-11 05:58:16.825 -0500] ERROR [p-8ec90860f6d8-DefaultBootstrapClusterJob] POST /api/v8/environments/HeatEnvironment2/deployments/Manager2/clusters com.cloudera.launchpad.boo
tstrap.cluster.DistributeParcel - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
java.lang.IllegalStateException: Parcel transition command skipped, but no parcels are in an intermediate stage
        at com.cloudera.launchpad.bootstrap.cluster.ParcelStageTransitionJob.run(ParcelStageTransitionJob.java:101)
        at com.cloudera.launchpad.bootstrap.cluster.ParcelStageTransitionJob.run(ParcelStageTransitionJob.java:38)
        at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
        at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
        at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
        at com.cloudera.launchpad.bootstrap.cluster.DistributeParcel$$EnhancerBySpringCGLIB$$f443e244.runUnchecked(<generated>)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2017-11-11 05:58:16.825 -0500] ERROR [p-8ec90860f6d8-DefaultBootstrapClusterJob] POST /api/v8/environments/HeatEnvironment2/deployments/Manager2/clusters com.cloudera.launchpad.bootstrap.cluster.DistributeParcel - c.c.l.p.DatabasePipelineRunner: Pipeline 2f097807-fc1b-4277-8273-8ec90860f6d8 suspended due to failure working on com.cloudera.launchpad.bootstrap.cluster.DistributeParcel
java.lang.IllegalStateException: Parcel transition command skipped, but no parcels are in an intermediate stage
        at com.cloudera.launchpad.bootstrap.cluster.ParcelStageTransitionJob.run(ParcelStageTransitionJob.java:101)

Any ideas?

/David

[#6 · Accepted solution]

Just to wrap up this thread, I managed to get the cluser up and running when selecting services as "Core hadoop" only instead of "Hadoop and Impala". Then some configuration to add Impala as a service manually got me to where I wanted.

/David