Member since
11-07-2017
5
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4154 | 11-12-2017 10:18 PM |
11-12-2017
10:18 PM
Just to wrap up this thread, I managed to get the cluser up and running when selecting services as "Core hadoop" only instead of "Hadoop and Impala". Then some configuration to add Impala as a service manually got me to where I wanted. /David
... View more
11-11-2017
03:13 AM
I have setup a local package and parcel repo but cluster bootstrap failing with: [2017-11-11 05:58:16.825 -0500] ERROR [p-8ec90860f6d8-DefaultBootstrapClusterJob] POST /api/v8/environments/HeatEnvironment2/deployments/Manager2/clusters com.cloudera.launchpad.boo
tstrap.cluster.DistributeParcel - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
java.lang.IllegalStateException: Parcel transition command skipped, but no parcels are in an intermediate stage
at com.cloudera.launchpad.bootstrap.cluster.ParcelStageTransitionJob.run(ParcelStageTransitionJob.java:101)
at com.cloudera.launchpad.bootstrap.cluster.ParcelStageTransitionJob.run(ParcelStageTransitionJob.java:38)
at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
at com.codahale.metrics.Timer.time(Timer.java:101)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.cluster.DistributeParcel$$EnhancerBySpringCGLIB$$f443e244.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2017-11-11 05:58:16.825 -0500] ERROR [p-8ec90860f6d8-DefaultBootstrapClusterJob] POST /api/v8/environments/HeatEnvironment2/deployments/Manager2/clusters com.cloudera.launchpad.bootstrap.cluster.DistributeParcel - c.c.l.p.DatabasePipelineRunner: Pipeline 2f097807-fc1b-4277-8273-8ec90860f6d8 suspended due to failure working on com.cloudera.launchpad.bootstrap.cluster.DistributeParcel
java.lang.IllegalStateException: Parcel transition command skipped, but no parcels are in an intermediate stage
at com.cloudera.launchpad.bootstrap.cluster.ParcelStageTransitionJob.run(ParcelStageTransitionJob.java:101)
Any ideas? /David
... View more
11-09-2017
10:53 PM
Thanks! I've now managed to setup a local package repo and got a Manager up and running through Director. Next problem is when adding a cluster, it fails with: Bootstrap failed Insufficient number of instances available in time 20 MINUTES [2017-11-10 00:56:39.555 -0500] INFO [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.director.aws.ec2.EC2Provider: Found EC2 key name heat-dev for fingerprint
[2017-11-10 00:56:39.560 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10081, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:39.568 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances/4 [Environment{name='Heat', provider=InstanceProviderConfig{type='aws'}, credentials=SshCredentials{us ...
[2017-11-10 00:56:39.568 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: All requested instances are available
[2017-11-10 00:56:39.568 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.bootstrap.AllocateInstances: Sufficient number of instances available (1/1)
[2017-11-10 00:56:39.584 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10082, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:39.594 -0500] INFO [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.director.aws.ec2.EC2Provider: >> Terminating [i-01e0d4a7f40831412, i-00dde88bcea48c247]
[2017-11-10 00:56:39.605 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$RefreshInstancesMetadata/2 [[PluggableComputeInstance{ipAddress=172.31.29.211, delegate=null, hostEndpoints=[HostEndpoint{hostA ...
[2017-11-10 00:56:39.715 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata - c.c.director.aws.ec2.EC2Provider: Found EC2 key name heat-dev for fingerprint
[2017-11-10 00:56:39.793 -0500] INFO [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.director.aws.ec2.EC2Provider: << Result {TerminatingInstances: [{InstanceId: i-00dde88bcea48c247,CurrentState: {Code: 32,Name: shutting-down},PreviousState: {Code: 16,Name: running}}, {InstanceId: i-01e0d4a7f40831412,CurrentState: {Code: 32,Name: shutting-down},PreviousState: {Code: 16,Name: running}}]}
[2017-11-10 00:56:39.794 -0500] ERROR [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307)
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261)
at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
at com.codahale.metrics.Timer.time(Timer.java:101)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$7ec0d65c.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2017-11-10 00:56:39.794 -0500] ERROR [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error ErrorInfo{code=ErrorCode{name=INSTANCE_ALLOCATION_TIME_OUT, type=SERVICE, retryable=true, keys=[minRequiredCount, timeoutInMinutes]}, properties={minRequiredCount=3, timeoutInMinutes=20}} in job com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available in time 20 MINUTES
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:307)
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances.run(AllocateInstances.java:261)
at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
at com.codahale.metrics.Timer.time(Timer.java:101)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$7ec0d65c.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:157)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2017-11-10 00:56:39.794 -0500] ERROR [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.DatabasePipelineRunner: Pipeline '856656a2-1693-4a66-b7ca-3a8e2a2de99b/child-00000-8b4b0060-a447-4b94-b2a0-ee24d88a4e8f' failed
at com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances$$EnhancerBySpringCGLIB$$7ec0d65c
at com.cloudera.launchpad.bootstrap.AllocateInstances.AllocateAndWaitForInstancesToRun:0
at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach:1
[2017-11-10 00:56:39.797 -0500] INFO [p-ee24d88a4e8f-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$GetSuccessfulInstancesAndTerminateFailedInstances - c.c.l.p.s.PipelineRepositoryService: Pipeline '856656a2-1693-4a66-b7ca-3a8e2a2de99b/child-00000-8b4b0060-a447-4b94-b2a0-ee24d88a4e8f': RUNNING -> ERROR
[2017-11-10 00:56:39.874 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata - c.c.director.aws.ec2.EC2Provider: >> Fetching page 0
[2017-11-10 00:56:39.908 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata - c.c.director.aws.ec2.EC2Provider: << Result: {InstanceStatuses: [{InstanceId: i-05b4b9efee75811c6,AvailabilityZone: cn-north-1a,Events: [],InstanceState: {Code: 16,Name: running},SystemStatus: {Status: initializing,Details: [{Name: reachability,Status: initializing,}]},InstanceStatus: {Status: initializing,Details: [{Name: reachability,Status: initializing,}]}}],}
[2017-11-10 00:56:39.919 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$RefreshInstancesMetadata - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10083, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:39.948 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForInstancePorts - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$WaitForInstancePorts/3 [[PluggableComputeInstance{ipAddress=172.31.29.211, delegate=null, hostEndpoints=[HostEndpoint{hostA ...
[2017-11-10 00:56:39.950 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForInstancePorts - c.c.l.bootstrap.AllocateInstances: Waiting for SSH port of instances to become available
[2017-11-10 00:56:39.962 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$WaitForInstancePorts - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10093, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:39.971 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches - c.c.l.pipeline.util.PipelineRunner: >> ParallelForEachInBatches/6 [20, class com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime, [PluggableComputeIn ...
[2017-11-10 00:56:39.977 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches - c.c.l.p.u.ParallelForEachInBatches: Generating batch for job class com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime of size 1
[2017-11-10 00:56:39.979 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10104, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:39.987 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach - c.c.l.pipeline.util.PipelineRunner: >> UnboundedParallelForEach/5 [class com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime, [PluggableComputeInstan ...
[2017-11-10 00:56:39.988 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach - c.c.l.p.DatabasePipelineService: Starting pipeline '856656a2-1693-4a66-b7ca-3a8e2a2de99b/child-00000-8d962b49-2ac6-4f05-a671-8523472863d1/child-00000-c36d6d33-e2d0-448d-a4e1-34ed3aaf0cef' with root job com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2017-11-10 00:56:39.999 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10114, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:40.003 -0500] INFO [p-8523472863d1-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.pipeline.util.UnboundedWaitForAllPipelines - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForAllPipelines/1 [[856656a2-1693-4a66-b7ca-3a8e2a2de99b/child-00000-8d962b49-2ac6-4f05-a671-8523472863d1/child-00000- ...
[2017-11-10 00:56:40.005 -0500] INFO [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime - c.c.l.pipeline.util.PipelineRunner: >> WaitForSshAccessOnInstanceUntilTime/4 [PluggableComputeInstance{ipAddress=172.31.29.211, delegate=null, hostEndpoints=[HostEndpoint{hostAd ...
[2017-11-10 00:56:40.017 -0500] INFO [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=10120, pipeline=856656a2-1693-4a66-b7ca-3a8e2a2de99b ...
[2017-11-10 00:56:40.023 -0500] INFO [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.pipeline.util.PipelineRunner: >> WaitForServersUntilTime/3 [[172.31.29.211:22, ip-172-31-29-211.cn-north-1.compute.internal:22, 52.80.58.19:22, ec2-52-80-58-19 ...
[2017-11-10 00:56:40.023 -0500] INFO [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Waiting for 1199927 milliseconds for an accessible port on endpoints [172.31.29.211:22, ip-172-31-29-211.cn-north-1.compute.internal:22, 52.80.58.19:22, ec2-52-80-58-19.cn-north-1.compute.amazonaws.com.cn:22]
[2017-11-10 00:56:40.023 -0500] INFO [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Checking every 10000 milliseconds
[2017-11-10 00:56:40.025 -0500] INFO [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /172.31.29.211:22
[2017-11-10 00:56:40.526 -0500] INFO [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to ip-172-31-29-211.cn-north-1.compute.internal/172.31.29.211:22
[2017-11-10 00:56:41.027 -0500] INFO [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /52.80.58.19:22
[2017-11-10 00:56:41.528 -0500] INFO [p-34ed3aaf0cef-WaitForSshAccessOnInstanceUntilTime] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to ec2-52-80-58-19.cn-north-1.compute.amazonaws.com.cn/52.80.58.19:22
[2017-11-10 00:56:43.701 -0500] INFO [p-b365a6e939c4-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: << Instance i-0094aa2d6b87403f0 got IP 172.31.21.72
[2017-11-10 00:56:43.727 -0500] INFO [p-b365a6e939c4-AllocateInstances] POST /api/v8/environments/Heat/deployments/HeatManager/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Waiting for 0 instances to start running
Its strange because I'm getting the above error just after like 1 minute, while the 5 ec2 instances are being initialized. They are actually eventually starting up ok as I can see in the ec2 console, but Director says failed! I can't seem to find the cause in the logfile? Director seems to both see the instances and think they are not finished and fails too early... /David
... View more
11-08-2017
05:17 AM
I was using Director 2.6, downgrading to 2.4 did not have the same problem. Next problem though: Seems to timeout when downloading yum packages, too slow connection from China. Does anyone know if using a proxy server works when installing using Cloudera Manager?
... View more
11-07-2017
10:52 PM
Hi, I've been struggling with installing Director in China region and finally managed to do it, but now I'm stuck when creating a Manager cluster. Cloudera Director fails when bootstrapping Manager. The ec2 instances are created but do not start completely, only error message I get is "Server.InternalError: Internal error on launch". /var/log/cloudera-director-server/application.log: [2017-11-08 06:39:20.659 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating deployment template: testmanager
[2017-11-08 06:39:20.659 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating parcel repository URLs
[2017-11-08 06:39:20.659 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating CSD URLs
[2017-11-08 06:39:20.659 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating external databases and templates
[2017-11-08 06:39:20.659 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.m.r.v.UsageCollectionHeartbeatDeploymentTemplateValidator: Validating deployment template: testmanager
[2017-11-08 06:39:20.659 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.p.c.PluggableComputeDeploymentTemplateValidator: Validating Cloudera Manager virtual instance template
[2017-11-08 06:39:20.659 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.p.c.PluggableComputeInstanceTemplateValidator: Validating instance template for compute provider: aws
[2017-11-08 06:39:20.797 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.director.aws.ec2.EC2Provider: Found EC2 key name heat-dev for fingerprint
[2017-11-08 06:39:20.797 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing AMI 'ami-3ce23651'
[2017-11-08 06:39:20.822 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing subnet 'subnet-f0b55994'
[2017-11-08 06:39:20.839 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing security group 'sg-4ec4aa2a'
[2017-11-08 06:39:20.860 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing key pair
[2017-11-08 06:39:20.928 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.p.DatabasePipelineService: Starting pipeline '696cc743-68c1-44fe-ac0c-f3c6eeaeb024' with root job com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob and listener com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener
[2017-11-08 06:39:20.933 +0000] INFO [qtp1378280450-1510] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '696cc743-68c1-44fe-ac0c-f3c6eeaeb024'
[2017-11-08 06:39:21.039 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments - - c.c.l.d.DeploymentRepositoryService: Deployment 'testmanager': BOOTSTRAPPING -> BOOTSTRAPPING
[2017-11-08 06:39:21.046 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob - c.c.l.pipeline.util.PipelineRunner: >> DefaultBootstrapDeploymentJob/2 [CreateDeploymentContext{environment=Environment{name='Heat', provider=InstanceProviderConfig{type=' ...
[2017-11-08 06:39:21.073 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=721, pipeline=696cc743-68c1-44fe-ac0c-f3c6eeaeb024, ...
[2017-11-08 06:39:21.079 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Requesting an instance for Cloudera Manager]
[2017-11-08 06:39:21.080 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Requesting an instance for Cloudera Manager
[2017-11-08 06:39:21.080 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-11-08 06:39:21.089 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances/2 [VirtualInstanceGroup{name='CM', virtualInstances=[VirtualInstance{id='63391619-bfcb-4b6f-9a87-f4af9 ...
[2017-11-08 06:39:21.102 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=730, pipeline=696cc743-68c1-44fe-ac0c-f3c6eeaeb024, ...
[2017-11-08 06:39:21.110 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$AllocateAndWaitForInstancesToRun/2 [VirtualInstanceGroup{name='CM', virtualInstances=[VirtualInstance{id='63391619-bfcb-4b6f-9a87-f4af9 ...
[2017-11-08 06:39:21.110 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Allocating 1 instances (min count 1) in group CM
[2017-11-08 06:39:21.191 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Found EC2 key name heat-dev for fingerprint
[2017-11-08 06:39:21.191 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Requesting 1 instances for com.cloudera.director.aws.ec2.EC2InstanceTemplate@7d85aff0
[2017-11-08 06:39:21.225 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Building 1 instance requests
[2017-11-08 06:39:21.225 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Network interface specification: {DeviceIndex: 0,SubnetId: subnet-f0b55994,Groups: [sg-4ec4aa2a],DeleteOnTermination: true,PrivateIpAddresses: [],AssociatePublicIpAddress: true,Ipv6Addresses: [],}
[2017-11-08 06:39:21.252 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Original image block device mappings: [{DeviceName: /dev/sda1,Ebs: {SnapshotId: snap-c8cfeefb,VolumeSize: 10,DeleteOnTermination: true,VolumeType: gp2,Encrypted: false},}]
[2017-11-08 06:39:21.252 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Block device mappings: [{DeviceName: /dev/sda1,Ebs: {SnapshotId: snap-c8cfeefb,VolumeSize: 50,DeleteOnTermination: true,VolumeType: gp2,},}]
[2017-11-08 06:39:21.252 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Instance request type: m4.xlarge, image: ami-3ce23651
[2017-11-08 06:39:21.252 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Submitted 1 run instance requests.
[2017-11-08 06:39:22.433 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: << Reservation r-0ac36602c7afb02f6 with Instance{id=i-0f418699fdc2412f8 privateIp=172.31.21.217}
[2017-11-08 06:39:22.697 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Retryer: resource not found, might be a transient error
com.cloudera.director.aws.shaded.com.amazonaws.AmazonServiceException: InvalidInstanceID.NotFound (Service: null; Status Code: 0; Error Code: InvalidInstanceID.NotFound; Request ID: null)
at com.cloudera.director.aws.ec2.EC2Provider$10.call(EC2Provider.java:1581)
at com.cloudera.director.aws.ec2.EC2Provider$10.call(EC2Provider.java:1)
at com.cloudera.director.aws.shaded.com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.cloudera.director.aws.shaded.com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.director.aws.ec2.EC2Retryer.retryUntil(EC2Retryer.java:94)
at com.cloudera.director.aws.ec2.EC2Retryer.retryUntil(EC2Retryer.java:81)
at com.cloudera.director.aws.ec2.EC2Retryer.retryUntil(EC2Retryer.java:71)
at com.cloudera.director.aws.ec2.EC2Provider.waitUntilInstanceHasStarted(EC2Provider.java:1588)
at com.cloudera.director.aws.ec2.EC2Provider.allocateOnDemandInstances(EC2Provider.java:1405)
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:725)
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:1)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:614)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:545)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:520)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:331)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:220)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:196)
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.GeneratedMethodAccessor228.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$604aec91.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:199)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:139)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2017-11-08 06:39:27.808 +0000] ERROR [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Instance i-0f418699fdc2412f8 has unexpectedly terminated, reason Server.InternalError: Internal error on launch
[2017-11-08 06:39:27.808 +0000] INFO [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: << Instance i-0f418699fdc2412f8 did not start.
[2017-11-08 06:39:27.808 +0000] ERROR [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Trying to allocate (1) instances, (0) instances allocated, below minimum count (1): (0) instances failed for instance limit exceeded, (0) instances failed for request limit exceeded, (1) instances failed when flipping from Pending to Running, including volume limit exceeded.
[2017-11-08 06:39:27.808 +0000] ERROR [p-f3c6eeaeb024-DefaultBootstrapDeploymentJob] 082655ae-872b-4e28-8ca8-43d04271240a POST /api/v10/environments/Heat/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Unsuccessful allocation of on demand instances. Terminating instances. I've tried different ami images etc. to no avail, any help is appreciated!
... View more
Labels:
- Labels:
-
Cloudera Manager