Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera Director Cluster Bootstrap Fails to install

Cloudera Director Cluster Bootstrap Fails to install

New Contributor

Hello,

 

I am having trouble with installing a cluster with Cloudera Director Bootstrap. The manager installs fine, however upon several attemps and troubleshooting I am unable to spin up the cluster.

 

Here is the tail from the installation, there is several pipeline failures.

 

I am using Redhat 6.7 ami. I am trying to spin up 1 master , 3 workers, 1 gateway. It fails each time.

 

Please advise.

 

 

[2016-06-07 10:59:11] INFO  [pipeline-thread-4] - c.c.l.p.s.PipelineRepositoryService: Pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb/child-00002-bc3263e4-bd51-45cb-8826-5764ebd0d942': RUNNING -> SUSPENDED
[2016-06-07 10:59:11] ERROR [pipeline-thread-2] - c.c.l.p.DatabasePipelineRunner: Pipeline 1761865e-96dd-4f24-8c94-df4fb63e88fb/child-00000-dcce903f-cb9b-4c63-b1ba-3ee26cd44879 suspended due to failure
com.cloudera.director.aws.shaded.com.amazonaws.AmazonServiceException: Security group sg-503cbb2b and subnet subnet-ac7231f4 belong to different networks. (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameter; Request ID: 5540e1f8-d72f-4dbb-a752-07216fd7d155)
	at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) ~[na:na]
	at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) ~[na:na]
	at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) ~[na:na]
	at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) ~[na:na]
	at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11806) ~[na:na]
	at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2Client.runInstances(AmazonEC2Client.java:5701) ~[na:na]
	at com.cloudera.director.aws.ec2.EC2Provider.allocateOnDemandInstances(EC2Provider.java:736) ~[na:na]
	at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:510) ~[na:na]
	at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:132) ~[na:na]
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:492) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:441) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:419) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:267) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:153) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:133) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$1cff5401.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
	at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
[2016-06-07 10:59:11] ERROR [pipeline-thread-2] - c.c.l.p.DatabasePipelineRunner: Pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb/child-00000-dcce903f-cb9b-4c63-b1ba-3ee26cd44879' failed
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$1cff5401
	at com.cloudera.launchpad.bootstrap.AllocateInstances:0
	at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach:0

[2016-06-07 10:59:11] INFO  [pipeline-thread-2] - c.c.l.p.s.PipelineRepositoryService: Pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb/child-00000-dcce903f-cb9b-4c63-b1ba-3ee26cd44879': RUNNING -> SUSPENDED
[2016-06-07 10:59:11] INFO  [pipeline-thread-1] - c.c.l.p.DatabasePipelineService: Interrupting runner thread for pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb/child-00000-dcce903f-cb9b-4c63-b1ba-3ee26cd44879'
[2016-06-07 10:59:11] INFO  [pipeline-thread-1] - c.c.l.p.DatabasePipelineService: Interrupting runner thread for pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb/child-00001-3049d81e-9347-4776-9d2f-841d6b9af363'
[2016-06-07 10:59:11] INFO  [pipeline-thread-1] - c.c.l.p.DatabasePipelineService: Interrupting runner thread for pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb/child-00002-bc3263e4-bd51-45cb-8826-5764ebd0d942'
[2016-06-07 10:59:11] ERROR [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Parallel execution of jobs failed. Total: 3 Canceled: 3 Failed: 0
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines.runUnchecked(UnboundedWaitForAllPipelines.java:71) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$FastClassBySpringCGLIB$$c9658ef0.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_91]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_91]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$EnhancerBySpringCGLIB$$173a2dd2.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
	at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
[2016-06-07 10:59:11] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Parallel execution of jobs failed. Total: 3 Canceled: 3 Failed: 0
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines.runUnchecked(UnboundedWaitForAllPipelines.java:71) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$FastClassBySpringCGLIB$$c9658ef0.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_91]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_91]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$EnhancerBySpringCGLIB$$173a2dd2.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
	at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
[2016-06-07 10:59:11] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb' failed
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$EnhancerBySpringCGLIB$$173a2dd2
	at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach:3

[2016-06-07 10:59:11] INFO  [pipeline-thread-3] - c.c.l.p.s.PipelineRepositoryService: Pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb/child-00001-3049d81e-9347-4776-9d2f-841d6b9af363': SUSPENDED -> CANCELED
[2016-06-07 10:59:11] INFO  [pipeline-thread-2] - c.c.l.p.s.PipelineRepositoryService: Pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb/child-00000-dcce903f-cb9b-4c63-b1ba-3ee26cd44879': SUSPENDED -> CANCELED
[2016-06-07 10:59:11] INFO  [pipeline-thread-4] - c.c.l.p.s.PipelineRepositoryService: Pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb/child-00002-bc3263e4-bd51-45cb-8826-5764ebd0d942': SUSPENDED -> CANCELED
[2016-06-07 10:59:11] INFO  [pipeline-thread-1] - c.c.l.p.s.PipelineRepositoryService: Pipeline '1761865e-96dd-4f24-8c94-df4fb63e88fb': RUNNING -> ERROR
[2016-06-07 10:59:11] INFO  [pipeline-thread-1] - c.c.l.d.ClusterRepositoryService: Cluster 'Pluto': BOOTSTRAPPING -> BOOTSTRAP_FAILED
7 REPLIES 7

Re: Cloudera Director Cluster Bootstrap Fails to install

Expert Contributor

I believe the first error is the root cause:
Security group sg-503cbb2b and subnet subnet-ac7231f4 belong to different networks.

 

Take a look at the securityGroupsIds and subnetId you have specified in your configuration file. The security groups and subnet need to belong to the same VPC.

 

Re: Cloudera Director Cluster Bootstrap Fails to install

New Contributor

Thanks for your reply.

 

I went back to using the instance template, the installation continues a bit further however it still failsat the end.

 

Here is the tail,

 

Thanks for your help.

 

[2016-06-07 12:17:46] INFO  [pipeline-thread-9] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-06-07 12:17:46] INFO  [pipeline-thread-9] - c.c.l.pipeline.util.PipelineRunner: >> WaitForAllValues/1 [null]
[2016-06-07 12:17:46] INFO  [pipeline-thread-9] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3205, pipeline=bbf1527d-4063-46fe-a48f-02299438e765/ ...
[2016-06-07 12:17:46] INFO  [pipeline-thread-9] - c.c.l.pipeline.util.PipelineRunner: >> CreateList/1 [[]]
[2016-06-07 12:17:46] INFO  [pipeline-thread-9] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3206, pipeline=bbf1527d-4063-46fe-a48f-02299438e765/ ...
[2016-06-07 12:17:46] INFO  [pipeline-thread-9] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'bbf1527d-4063-46fe-a48f-02299438e765/child-00002-389b016b-2509-43a6-9cf6-8327930c8593': RUNNING -> COMPLETED
[2016-06-07 12:17:46] INFO  [pipeline-thread-6] - c.c.l.p.DatabasePipelineService: Interrupting runner thread for pipeline 'bbf1527d-4063-46fe-a48f-02299438e765/child-00000-fde4c975-0cf7-416b-96f7-09ba363e36ca'
[2016-06-07 12:17:46] INFO  [pipeline-thread-6] - c.c.l.p.DatabasePipelineService: Interrupting runner thread for pipeline 'bbf1527d-4063-46fe-a48f-02299438e765/child-00001-9fc458cf-31f8-41bb-b146-3eb5e00715f9'
[2016-06-07 12:17:46] ERROR [pipeline-thread-6] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Parallel execution of jobs failed. Total: 3 Canceled: 2 Failed: 0
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines.runUnchecked(UnboundedWaitForAllPipelines.java:71) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$FastClassBySpringCGLIB$$c9658ef0.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$EnhancerBySpringCGLIB$$173a2dd2.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
	at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
[2016-06-07 12:17:46] INFO  [pipeline-thread-7] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'bbf1527d-4063-46fe-a48f-02299438e765/child-00000-fde4c975-0cf7-416b-96f7-09ba363e36ca': SUSPENDED -> CANCELED
[2016-06-07 12:17:46] ERROR [pipeline-thread-6] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Parallel execution of jobs failed. Total: 3 Canceled: 2 Failed: 0
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines.runUnchecked(UnboundedWaitForAllPipelines.java:71) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$FastClassBySpringCGLIB$$c9658ef0.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$EnhancerBySpringCGLIB$$173a2dd2.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
	at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
[2016-06-07 12:17:46] ERROR [pipeline-thread-6] - c.c.l.p.DatabasePipelineRunner: Pipeline 'bbf1527d-4063-46fe-a48f-02299438e765' failed
	at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$EnhancerBySpringCGLIB$$173a2dd2
	at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach:3

[2016-06-07 12:17:46] INFO  [pipeline-thread-6] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'bbf1527d-4063-46fe-a48f-02299438e765': RUNNING -> ERROR
[2016-06-07 12:17:46] INFO  [pipeline-thread-8] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'bbf1527d-4063-46fe-a48f-02299438e765/child-00001-9fc458cf-31f8-41bb-b146-3eb5e00715f9': SUSPENDED -> CANCELED
[2016-06-07 12:17:46] INFO  [pipeline-thread-6] - c.c.l.d.ClusterRepositoryService: Cluster 'Ckram': BOOTSTRAPPING -> BOOTSTRAP_FAILED

Re: Cloudera Director Cluster Bootstrap Fails to install

Expert Contributor

Unfortunately, those are high-level errors that do not indicate the root cause. Do you have more of the tail available?

 

Re: Cloudera Director Cluster Bootstrap Fails to install

New Contributor

See below.

Re: Cloudera Director Cluster Bootstrap Fails to install

New Contributor

[2016-06-07 12:40:00] INFO [pipeline-thread-26] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.135:22
[2016-06-07 12:40:01] INFO [pipeline-thread-25] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.36:22
[2016-06-07 12:40:11] INFO [pipeline-thread-26] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.135:22
[2016-06-07 12:40:11] INFO [pipeline-thread-25] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.36:22
[2016-06-07 12:40:21] INFO [pipeline-thread-26] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.135:22
[2016-06-07 12:40:22] INFO [pipeline-thread-25] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.36:22
[2016-06-07 12:40:29] INFO [pipeline-thread-23] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$AllocateAndWaitForInstancesToRun/2 [VirtualInstanceGroup{name='workers', virtualInstances=[VirtualInstance{id='fb46d451-7a0c-4bd7-b169- ...
[2016-06-07 12:40:29] INFO [pipeline-thread-23] - c.c.l.bootstrap.AllocateInstances: Allocating 3 instances (min count 3) in group workers
[2016-06-07 12:40:29] INFO [pipeline-thread-23] - c.c.director.aws.ec2.EC2Provider: Found EC2 key name aws-cloudera-quickstart-06-03-2016 for fingerprint
[2016-06-07 12:40:29] INFO [pipeline-thread-23] - c.c.director.aws.ec2.EC2Provider: >> Requesting 3 instances for com.cloudera.director.aws.ec2.EC2InstanceTemplate@2352a61a
[2016-06-07 12:40:29] INFO [pipeline-thread-23] - c.c.director.aws.ec2.EC2Provider: >> Network interface specification: {DeviceIndex: 0,SubnetId: subnet-af7231f7,Groups: [sg-828c0af9],DeleteOnTermination: true,PrivateIpAddresses: [],AssociatePublicIpAddress: true}
[2016-06-07 12:40:31] INFO [pipeline-thread-23] - c.c.director.aws.ec2.EC2Provider: >> Block device mappings: [{DeviceName: /dev/sda1,Ebs: {SnapshotId: snap-8d1009f8,VolumeSize: 50,DeleteOnTermination: true,VolumeType: gp2,},}, {VirtualName: ephemeral0,DeviceName: /dev/sdb,}, {VirtualName: ephemeral1,DeviceName: /dev/sdc,}]
[2016-06-07 12:40:32] ERROR [pipeline-thread-23] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.director.aws.shaded.com.amazonaws.AmazonServiceException: You have requested more instances (5) than your current instance limit of 1 allows for the specified instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit. (Service: AmazonEC2; Status Code: 400; Error Code: InstanceLimitExceeded; Request ID: 87cd41d3-f928-481a-b61b-8f9a751be340)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) ~[na:na]
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) ~[na:na]
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) ~[na:na]
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) ~[na:na]
at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11806) ~[na:na]
at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2Client.runInstances(AmazonEC2Client.java:5701) ~[na:na]
at com.cloudera.director.aws.ec2.EC2Provider.allocateOnDemandInstances(EC2Provider.java:736) ~[na:na]
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:510) ~[na:na]
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:132) ~[na:na]
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:492) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:441) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:419) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:267) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:153) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:133) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$1cff5401.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
[2016-06-07 12:40:32] ERROR [pipeline-thread-23] - c.c.l.p.DatabasePipelineRunner: Pipeline b7afe6e4-2376-4434-907e-7736e0a3c096/child-00001-5818d68b-0ce6-4615-8129-632c1d8b62f8 suspended due to failure
com.cloudera.director.aws.shaded.com.amazonaws.AmazonServiceException: You have requested more instances (5) than your current instance limit of 1 allows for the specified instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit. (Service: AmazonEC2; Status Code: 400; Error Code: InstanceLimitExceeded; Request ID: 87cd41d3-f928-481a-b61b-8f9a751be340)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) ~[na:na]
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) ~[na:na]
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) ~[na:na]
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) ~[na:na]
at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11806) ~[na:na]
at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2Client.runInstances(AmazonEC2Client.java:5701) ~[na:na]
at com.cloudera.director.aws.ec2.EC2Provider.allocateOnDemandInstances(EC2Provider.java:736) ~[na:na]
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:510) ~[na:na]
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:132) ~[na:na]
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:492) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:441) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:419) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:267) ~[launchpad-pluggable-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:153) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:133) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$1cff5401.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
[2016-06-07 12:40:32] ERROR [pipeline-thread-23] - c.c.l.p.DatabasePipelineRunner: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00001-5818d68b-0ce6-4615-8129-632c1d8b62f8' failed
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$1cff5401
at com.cloudera.launchpad.bootstrap.AllocateInstances:0
at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach:1

[2016-06-07 12:40:32] INFO [pipeline-thread-23] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00001-5818d68b-0ce6-4615-8129-632c1d8b62f8': RUNNING -> SUSPENDED
[2016-06-07 12:40:32] INFO [pipeline-thread-26] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.135:22
[2016-06-07 12:40:32] INFO [pipeline-thread-25] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.36:22
[2016-06-07 12:40:42] INFO [pipeline-thread-26] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.135:22
[2016-06-07 12:40:42] INFO [pipeline-thread-25] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.36:22
[2016-06-07 12:40:52] INFO [pipeline-thread-26] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.135:22
[2016-06-07 12:40:52] INFO [pipeline-thread-25] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.36:22
[2016-06-07 12:41:02] INFO [pipeline-thread-26] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.135:22
[2016-06-07 12:41:02] INFO [pipeline-thread-25] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.36:22
[2016-06-07 12:41:12] INFO [pipeline-thread-26] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.135:22
[2016-06-07 12:41:12] INFO [pipeline-thread-25] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.36:22
[2016-06-07 12:41:22] INFO [pipeline-thread-26] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.135:22
[2016-06-07 12:41:22] INFO [pipeline-thread-26] - c.c.l.b.WaitForServerOnPortUntilTime: Successfully found server 10.0.1.135 listening at /10.0.1.135:22
[2016-06-07 12:41:22] INFO [pipeline-thread-26] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3675, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-26] - c.c.l.pipeline.util.PipelineRunner: >> WaitForInstanceOnPortUntilTime$FailedInstanceOrNone/2 [PluggableComputeInstance{ipAddress=Optional.of(10.0.1.135), delegate=null} Instance{virtualInstance ...
[2016-06-07 12:41:22] INFO [pipeline-thread-26] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-06-07 12:41:22] INFO [pipeline-thread-26] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00002-7c2cdab8-5397-4b7c-afa8-cfee588c4a52/child-00000-96116f3e-fc39-4ab8-ab92-52946637c869': RUNNING -> COMPLETED
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: >> WaitForAllValues/1 [null]
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3676, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: >> CreateList/1 [[]]
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3677, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances/4 [Environment{name='C5-Reference-AWS Environment', provider=InstanceProviderConfig{type='aws'}, crede ...
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.bootstrap.AllocateInstances: All requested instances are available
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.bootstrap.AllocateInstances: Sufficient number of instances available (1/1)
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3678, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: >> ParallelForEachInBatches/4 [20, class com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForSshCredentialInstallation, [Plu ...
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.p.u.ParallelForEachInBatches: Generating batch for job class com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForSshCredentialInstallation of size 1
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3683, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedParallelForEach/3 [class com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForSshCredentialInstallation, [Pluggab ...
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.p.DatabasePipelineService: Starting pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00002-7c2cdab8-5397-4b7c-afa8-cfee588c4a52/child-00000-28e806fe-52f5-417d-8e64-6afb65f5c0c0' with root job com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForSshCredentialInstallation and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3689, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-26] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$WaitForSshCredentialInstallation/2 [PluggableComputeInstance{ipAddress=Optional.of(10.0.1.135), delegate=null} Instance{virtualInstance ...
[2016-06-07 12:41:22] INFO [pipeline-thread-26] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForAllPipelines/1 [b7afe6e4-2376-4434-907e-7736e0a3c096/child-00002-7c2cdab8-5397-4b7c-afa8-cfee588c4a52/child-00000-2 ...
[2016-06-07 12:41:22] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.1.135 was automatically accepted
[2016-06-07 12:41:22] INFO [pipeline-thread-26] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-06-07 12:41:22] INFO [pipeline-thread-26] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00002-7c2cdab8-5397-4b7c-afa8-cfee588c4a52/child-00000-28e806fe-52f5-417d-8e64-6afb65f5c0c0': RUNNING -> COMPLETED
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: >> WaitForAllValues/1 [null]
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3690, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: >> CreateList/1 [[]]
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3691, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-24] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00002-7c2cdab8-5397-4b7c-afa8-cfee588c4a52': RUNNING -> COMPLETED
[2016-06-07 12:41:22] INFO [pipeline-thread-25] - c.c.l.b.WaitForServerOnPortUntilTime: Attempting connection to /10.0.1.36:22
[2016-06-07 12:41:22] INFO [pipeline-thread-25] - c.c.l.b.WaitForServerOnPortUntilTime: Successfully found server 10.0.1.36 listening at /10.0.1.36:22
[2016-06-07 12:41:22] INFO [pipeline-thread-25] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3692, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-25] - c.c.l.pipeline.util.PipelineRunner: >> WaitForInstanceOnPortUntilTime$FailedInstanceOrNone/2 [PluggableComputeInstance{ipAddress=Optional.of(10.0.1.36), delegate=null} Instance{virtualInstance= ...
[2016-06-07 12:41:22] INFO [pipeline-thread-25] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-06-07 12:41:22] INFO [pipeline-thread-25] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00000-34096564-3d70-4e90-9e68-2b1a4422ad1a/child-00000-fc7beef2-3004-4af2-aacc-1dd374408cf6': RUNNING -> COMPLETED
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: >> WaitForAllValues/1 [null]
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3693, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: >> CreateList/1 [[]]
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3694, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances/4 [Environment{name='C5-Reference-AWS Environment', provider=InstanceProviderConfig{type='aws'}, crede ...
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.bootstrap.AllocateInstances: All requested instances are available
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.bootstrap.AllocateInstances: Sufficient number of instances available (1/1)
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3695, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: >> ParallelForEachInBatches/4 [20, class com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForSshCredentialInstallation, [Plu ...
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.p.u.ParallelForEachInBatches: Generating batch for job class com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForSshCredentialInstallation of size 1
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3700, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedParallelForEach/3 [class com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForSshCredentialInstallation, [Pluggab ...
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.p.DatabasePipelineService: Starting pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00000-34096564-3d70-4e90-9e68-2b1a4422ad1a/child-00000-b7b60ea8-60f7-4061-a844-effaca03fcd3' with root job com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForSshCredentialInstallation and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3706, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:22] INFO [pipeline-thread-25] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$WaitForSshCredentialInstallation/2 [PluggableComputeInstance{ipAddress=Optional.of(10.0.1.36), delegate=null} Instance{virtualInstance= ...
[2016-06-07 12:41:22] INFO [pipeline-thread-25] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2016-06-07 12:41:22] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForAllPipelines/1 [b7afe6e4-2376-4434-907e-7736e0a3c096/child-00000-34096564-3d70-4e90-9e68-2b1a4422ad1a/child-00000-b ...
[2016-06-07 12:41:22] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.1.36 was automatically accepted
[2016-06-07 12:41:23] INFO [pipeline-thread-25] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-06-07 12:41:23] INFO [pipeline-thread-25] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00000-34096564-3d70-4e90-9e68-2b1a4422ad1a/child-00000-b7b60ea8-60f7-4061-a844-effaca03fcd3': RUNNING -> COMPLETED
[2016-06-07 12:41:23] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2016-06-07 12:41:23] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: >> WaitForAllValues/1 [null]
[2016-06-07 12:41:23] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3707, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:23] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: >> CreateList/1 [[]]
[2016-06-07 12:41:23] INFO [pipeline-thread-22] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=3708, pipeline=b7afe6e4-2376-4434-907e-7736e0a3c096/ ...
[2016-06-07 12:41:23] INFO [pipeline-thread-22] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00000-34096564-3d70-4e90-9e68-2b1a4422ad1a': RUNNING -> COMPLETED
[2016-06-07 12:41:23] INFO [pipeline-thread-17] - c.c.l.p.DatabasePipelineService: Interrupting runner thread for pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00001-5818d68b-0ce6-4615-8129-632c1d8b62f8'
[2016-06-07 12:41:23] INFO [pipeline-thread-23] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096/child-00001-5818d68b-0ce6-4615-8129-632c1d8b62f8': SUSPENDED -> CANCELED
[2016-06-07 12:41:23] ERROR [pipeline-thread-17] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Parallel execution of jobs failed. Total: 3 Canceled: 1 Failed: 0
at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines.runUnchecked(UnboundedWaitForAllPipelines.java:71) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$FastClassBySpringCGLIB$$c9658ef0.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$EnhancerBySpringCGLIB$$173a2dd2.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
[2016-06-07 12:41:23] ERROR [pipeline-thread-17] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Parallel execution of jobs failed. Total: 3 Canceled: 1 Failed: 0
at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines.runUnchecked(UnboundedWaitForAllPipelines.java:71) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$FastClassBySpringCGLIB$$c9658ef0.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$EnhancerBySpringCGLIB$$173a2dd2.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
[2016-06-07 12:41:23] ERROR [pipeline-thread-17] - c.c.l.p.DatabasePipelineRunner: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096' failed
at com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines$$EnhancerBySpringCGLIB$$173a2dd2
at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach:3

[2016-06-07 12:41:23] INFO [pipeline-thread-17] - c.c.l.p.s.PipelineRepositoryService: Pipeline 'b7afe6e4-2376-4434-907e-7736e0a3c096': RUNNING -> ERROR
[2016-06-07 12:41:23] INFO [pipeline-thread-17] - c.c.l.d.ClusterRepositoryService: Cluster 'Cluster M': BOOTSTRAPPING -> BOOTSTRAP_FAILED

Re: Cloudera Director Cluster Bootstrap Fails to install

New Contributor

I believe I found the error in the begining of the code

 

You have requested more instances (5) than your current instance limit of 1 allows for the specified instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.

 

I will write back once succefully installed.

 

Thanks.

Re: Cloudera Director Cluster Bootstrap Fails to install

Expert Contributor

Yes, that is the root cause of the final set of failures.