Support Questions
Find answers, ask questions, and share your expertise

Cluster failed to bootstrap on cloudera manager

New Contributor

I'm following this tutorial on how to get up and running with Cloudera on Google Cloud Compute. I'm on this part of the tutorial for a reference. 

 

I added a manager but I used an 'n1-standard-1' instance since I'm only trying the platform out and not actually running it in a production environment. I kept following the tutorial until I got to the 'add a cluster' section. I changed the default values and I only have (1) master (1) worker and (1) gateway all configured to be an 'n1-standard-1' instance. I selected on the basic hadoop installation and nothing more.I hit continue and my cluster failed to bootstrap. This is what shows up on my screen.

 

Screenshot (49).png

 

 I checked the logs and this is what I found.

[2017-02-22 13:39:04.235 +0000] INFO  [qtp61426020-17] - c.c.l.p.c.PluggableComputeClusterTemplateValidator: Validating virtual instances of cluster TRIDENT_MEME
[2017-02-22 13:39:04.235 +0000] INFO  [qtp61426020-17] - c.c.l.p.c.PluggableComputeInstanceTemplateValidator: Validating instance template for compute provider: google
[2017-02-22 13:39:04.482 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeProviderConfigurationValidator: >> Querying region 'us-central1'
[2017-02-22 13:39:04.657 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying zone 'us-central1-a'
[2017-02-22 13:39:04.768 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying image 'centos6'
[2017-02-22 13:39:04.879 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying machine type 'n1-standard-1'
[2017-02-22 13:39:04.942 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying network 'default'
[2017-02-22 13:39:05.049 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2017-02-22 13:39:05.050 +0000] INFO  [qtp61426020-17] - c.c.l.p.c.PluggableComputeInstanceTemplateValidator: Validating instance template for compute provider: google
[2017-02-22 13:39:05.050 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeProviderConfigurationValidator: >> Querying region 'us-central1'
[2017-02-22 13:39:05.249 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying zone 'us-central1-a'
[2017-02-22 13:39:05.306 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying image 'centos6'
[2017-02-22 13:39:05.409 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying machine type 'n1-standard-1'
[2017-02-22 13:39:05.490 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying network 'default'
[2017-02-22 13:39:05.558 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2017-02-22 13:39:05.558 +0000] INFO  [qtp61426020-17] - c.c.l.p.c.PluggableComputeInstanceTemplateValidator: Validating instance template for compute provider: google
[2017-02-22 13:39:05.558 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeProviderConfigurationValidator: >> Querying region 'us-central1'
[2017-02-22 13:39:05.738 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying zone 'us-central1-a'
[2017-02-22 13:39:05.813 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying image 'centos6'
[2017-02-22 13:39:05.928 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying machine type 'n1-standard-1'
[2017-02-22 13:39:05.990 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Querying network 'default'
[2017-02-22 13:39:06.070 +0000] INFO  [qtp61426020-17] - c.c.d.g.c.GoogleComputeInstanceTemplateConfigurationValidator: >> Validating prefix 'director'
[2017-02-22 13:39:06.074 +0000] INFO  [qtp61426020-17] - c.c.l.p.DatabasePipelineService: Starting pipeline '5d013da5-b8dc-4900-85e1-f3f9845605b9' with root job com.cloudera.launchpad.api.jobs.DefaultBootstrapClusterJob and listener com.cloudera.launchpad.api.listeners.pipeline.BootstrapClusterListener
[2017-02-22 13:39:06.093 +0000] INFO  [pipeline-thread-17] - c.c.l.d.ClusterRepositoryService: Cluster 'TRIDENT_MEME': BOOTSTRAPPING -> BOOTSTRAPPING
[2017-02-22 13:39:06.123 +0000] INFO  [pipeline-thread-17] - c.c.l.pipeline.util.PipelineRunner: >> DefaultBootstrapClusterJob/4 [Environment{name='training-env', provider=InstanceProviderConfig{type='google'}, credentials=SshCre ...
[2017-02-22 13:39:06.372 +0000] INFO  [pipeline-thread-17] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=1509, pipeline=5d013da5-b8dc-4900-85e1-f3f9845605b9, ...
[2017-02-22 13:39:06.375 +0000] INFO  [pipeline-thread-17] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Requesting 3 instance(s) in 3 group(s)]
[2017-02-22 13:39:06.376 +0000] INFO  [pipeline-thread-17] - c.c.launchpad.pipeline.AbstractJob: Requesting 3 instance(s) in 3 group(s)
[2017-02-22 13:39:06.376 +0000] INFO  [pipeline-thread-17] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-02-22 13:39:06.392 +0000] INFO  [pipeline-thread-17] - c.c.l.pipeline.util.PipelineRunner: >> ParallelForEachInBatches/4 [20, class com.cloudera.launchpad.bootstrap.AllocateInstances, [VirtualInstanceGroup{name='masters', ...
[2017-02-22 13:39:06.400 +0000] INFO  [pipeline-thread-17] - c.c.l.p.u.ParallelForEachInBatches: Generating batch for job class com.cloudera.launchpad.bootstrap.AllocateInstances of size 3
[2017-02-22 13:39:06.402 +0000] INFO  [pipeline-thread-17] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=1514, pipeline=5d013da5-b8dc-4900-85e1-f3f9845605b9, ...
[2017-02-22 13:39:06.423 +0000] INFO  [pipeline-thread-17] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedParallelForEach/3 [class com.cloudera.launchpad.bootstrap.AllocateInstances, [VirtualInstanceGroup{name='masters', vir ...
[2017-02-22 13:39:06.424 +0000] INFO  [pipeline-thread-17] - c.c.l.p.DatabasePipelineService: Starting pipeline '5d013da5-b8dc-4900-85e1-f3f9845605b9/child-00000-ddbe1718-937b-4302-849f-a94fe8ecd3d9' with root job com.cloudera.launchpad.bootstrap.AllocateInstances and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2017-02-22 13:39:06.442 +0000] INFO  [pipeline-thread-17] - c.c.l.p.DatabasePipelineService: Starting pipeline '5d013da5-b8dc-4900-85e1-f3f9845605b9/child-00000-7152ab9e-560c-449a-a47a-963db8d530ff' with root job com.cloudera.launchpad.bootstrap.AllocateInstances and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2017-02-22 13:39:06.464 +0000] INFO  [pipeline-thread-17] - c.c.l.p.DatabasePipelineService: Starting pipeline '5d013da5-b8dc-4900-85e1-f3f9845605b9/child-00000-c87cd5fe-375a-4305-80c9-15db26892c97' with root job com.cloudera.launchpad.bootstrap.AllocateInstances and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener
[2017-02-22 13:39:06.491 +0000] INFO  [pipeline-thread-17] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=1528, pipeline=5d013da5-b8dc-4900-85e1-f3f9845605b9, ...
[2017-02-22 13:39:06.532 +0000] INFO  [pipeline-thread-18] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances/2 [VirtualInstanceGroup{name='masters', virtualInstances=[VirtualInstance{id='f2629f7b-8a82-4d79-b595- ...
[2017-02-22 13:39:06.542 +0000] INFO  [pipeline-thread-19] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances/2 [VirtualInstanceGroup{name='workers', virtualInstances=[VirtualInstance{id='b015d7f8-9a4e-4279-a824- ...
[2017-02-22 13:39:06.554 +0000] INFO  [pipeline-thread-17] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForAllPipelines/1 [[5d013da5-b8dc-4900-85e1-f3f9845605b9/child-00000-ddbe1718-937b-4302-849f-a94fe8ecd3d9, 5d013da5-b8 ...
[2017-02-22 13:39:06.564 +0000] INFO  [pipeline-thread-20] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances/2 [VirtualInstanceGroup{name='gateway', virtualInstances=[VirtualInstance{id='94dc5f62-45f3-45c1-882e- ...
[2017-02-22 13:39:06.576 +0000] INFO  [pipeline-thread-20] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=1533, pipeline=5d013da5-b8dc-4900-85e1-f3f9845605b9/ ...
[2017-02-22 13:39:06.585 +0000] INFO  [pipeline-thread-18] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=1538, pipeline=5d013da5-b8dc-4900-85e1-f3f9845605b9/ ...
[2017-02-22 13:39:06.594 +0000] INFO  [pipeline-thread-19] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=1543, pipeline=5d013da5-b8dc-4900-85e1-f3f9845605b9/ ...
[2017-02-22 13:39:06.662 +0000] INFO  [pipeline-thread-19] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$AllocateAndWaitForInstancesToRun/2 [VirtualInstanceGroup{name='workers', virtualInstances=[VirtualInstance{id='b015d7f8-9a4e-4279-a824- ...
[2017-02-22 13:39:06.662 +0000] INFO  [pipeline-thread-19] - c.c.l.bootstrap.AllocateInstances: Allocating 1 instances (min count 1) in group workers
[2017-02-22 13:39:06.693 +0000] INFO  [pipeline-thread-18] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$AllocateAndWaitForInstancesToRun/2 [VirtualInstanceGroup{name='masters', virtualInstances=[VirtualInstance{id='f2629f7b-8a82-4d79-b595- ...
[2017-02-22 13:39:06.693 +0000] INFO  [pipeline-thread-18] - c.c.l.bootstrap.AllocateInstances: Allocating 1 instances (min count 1) in group masters
[2017-02-22 13:39:06.705 +0000] INFO  [pipeline-thread-20] - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$AllocateAndWaitForInstancesToRun/2 [VirtualInstanceGroup{name='gateway', virtualInstances=[VirtualInstance{id='94dc5f62-45f3-45c1-882e- ...
[2017-02-22 13:39:06.706 +0000] INFO  [pipeline-thread-20] - c.c.l.bootstrap.AllocateInstances: Allocating 1 instances (min count 1) in group gateway
[2017-02-22 13:39:11.923 +0000] INFO  [pipeline-thread-20] - c.c.d.g.c.GoogleComputeProvider: Provisioned 0 instances out of 1. minCount is 1. Tearing down provisioned instances.
[2017-02-22 13:39:11.947 +0000] INFO  [pipeline-thread-19] - c.c.d.g.c.GoogleComputeProvider: Provisioned 0 instances out of 1. minCount is 1. Tearing down provisioned instances.
[2017-02-22 13:39:11.990 +0000] INFO  [pipeline-thread-18] - c.c.d.g.c.GoogleComputeProvider: Provisioned 0 instances out of 1. minCount is 1. Tearing down provisioned instances.
[2017-02-22 13:39:14.544 +0000] ERROR [pipeline-thread-20] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Problem allocating instances.
        at com.cloudera.launchpad.pluggable.common.ExceptionConditions.propagateWithSuppressedDetails(ExceptionConditions.java:114)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:534)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:480)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:458)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:306)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:154)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:134)
        at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
        at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
        at sun.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$c8e98257.runUnchecked(<generated>)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:164)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:135)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:104)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:128)
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Suppressed: com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:
  key: null
    PluginExceptionCondition{type=ERROR, message='Quota 'DISKS_TOTAL_GB' exceeded.  Limit: 2048.0'}

Caused by: com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException: Problem allocating instances.
        at com.cloudera.director.google.compute.GoogleComputeProvider.allocate(GoogleComputeProvider.java:401)
        at com.cloudera.director.google.compute.GoogleComputeProvider.allocate(GoogleComputeProvider.java:86)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:531)
        ... 37 common frames omitted
[2017-02-22 13:39:14.544 +0000] ERROR [pipeline-thread-20] - c.c.l.pipeline.util.PipelineRunner: (Suppressed)
com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:
  key: null
    PluginExceptionCondition{type=ERROR, message='Quota 'DISKS_TOTAL_GB' exceeded.  Limit: 2048.0'}

[2017-02-22 13:39:14.544 +0000] ERROR [pipeline-thread-20] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Problem allocating instances.
        at com.cloudera.launchpad.pluggable.common.ExceptionConditions.propagateWithSuppressedDetails(ExceptionConditions.java:114)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:534)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:480)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:458)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:306)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:154)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:134)
        at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
        at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
        at sun.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$c8e98257.runUnchecked(<generated>)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:164)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:135)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:104)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:128)
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Suppressed: com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:
  key: null
    PluginExceptionCondition{type=ERROR, message='Quota 'DISKS_TOTAL_GB' exceeded.  Limit: 2048.0'}

Caused by: com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException: Problem allocating instances.
        at com.cloudera.director.google.compute.GoogleComputeProvider.allocate(GoogleComputeProvider.java:401)
        at com.cloudera.director.google.compute.GoogleComputeProvider.allocate(GoogleComputeProvider.java:86)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:531)
        ... 37 common frames omitted
[2017-02-22 13:39:14.544 +0000] ERROR [pipeline-thread-20] - c.c.l.p.DatabasePipelineRunner: (Suppressed)
com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:
  key: null
    PluginExceptionCondition{type=ERROR, message='Quota 'DISKS_TOTAL_GB' exceeded.  Limit: 2048.0'}

[2017-02-22 13:39:14.545 +0000] ERROR [pipeline-thread-20] - c.c.l.p.DatabasePipelineRunner: Pipeline '5d013da5-b8dc-4900-85e1-f3f9845605b9/child-00000-c87cd5fe-375a-4305-80c9-15db26892c97' failed
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$c8e98257
        at com.cloudera.launchpad.bootstrap.AllocateInstances:0
        at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach:2

[2017-02-22 13:39:14.549 +0000] INFO  [pipeline-thread-20] - c.c.l.p.s.PipelineRepositoryService: Pipeline '5d013da5-b8dc-4900-85e1-f3f9845605b9/child-00000-c87cd5fe-375a-4305-80c9-15db26892c97': RUNNING -> ERROR
[2017-02-22 13:39:14.613 +0000] ERROR [pipeline-thread-18] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Problem allocating instances.
        at com.cloudera.launchpad.pluggable.common.ExceptionConditions.propagateWithSuppressedDetails(ExceptionConditions.java:114)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:534)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:480)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:458)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:306)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:154)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:134)
        at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
        at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
        at sun.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$c8e98257.runUnchecked(<generated>)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:164)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:135)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:104)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:128)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Suppressed: com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:
  key: null
    PluginExceptionCondition{type=ERROR, message='Quota 'DISKS_TOTAL_GB' exceeded.  Limit: 2048.0'}

Caused by: com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException: Problem allocating instances.
        at com.cloudera.director.google.compute.GoogleComputeProvider.allocate(GoogleComputeProvider.java:401)
        at com.cloudera.director.google.compute.GoogleComputeProvider.allocate(GoogleComputeProvider.java:86)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:531)
        ... 37 common frames omitted
[2017-02-22 13:39:14.614 +0000] ERROR [pipeline-thread-18] - c.c.l.pipeline.util.PipelineRunner: (Suppressed)
com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:
  key: null
    PluginExceptionCondition{type=ERROR, message='Quota 'DISKS_TOTAL_GB' exceeded.  Limit: 2048.0'}

[2017-02-22 13:39:14.614 +0000] ERROR [pipeline-thread-18] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Problem allocating instances.
        at com.cloudera.launchpad.pluggable.common.ExceptionConditions.propagateWithSuppressedDetails(ExceptionConditions.java:114)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:534)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:480)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:458)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:306)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:154)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:134)
        at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
        at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
        at sun.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$c8e98257.runUnchecked(<generated>)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:164)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:135)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:104)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:128)
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Suppressed: com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:
  key: null
    PluginExceptionCondition{type=ERROR, message='Quota 'DISKS_TOTAL_GB' exceeded.  Limit: 2048.0'}

Caused by: com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException: Problem allocating instances.
        at com.cloudera.director.google.compute.GoogleComputeProvider.allocate(GoogleComputeProvider.java:401)
        at com.cloudera.director.google.compute.GoogleComputeProvider.allocate(GoogleComputeProvider.java:86)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:531)
        ... 37 common frames omitted
[2017-02-22 13:39:14.614 +0000] ERROR [pipeline-thread-18] - c.c.l.p.DatabasePipelineRunner: (Suppressed)
com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:
  key: null
    PluginExceptionCondition{type=ERROR, message='Quota 'DISKS_TOTAL_GB' exceeded.  Limit: 2048.0'}

[2017-02-22 13:39:14.614 +0000] ERROR [pipeline-thread-18] - c.c.l.p.DatabasePipelineRunner: Pipeline '5d013da5-b8dc-4900-85e1-f3f9845605b9/child-00000-ddbe1718-937b-4302-849f-a94fe8ecd3d9' failed
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$c8e98257
        at com.cloudera.launchpad.bootstrap.AllocateInstances:0
        at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach:0

[2017-02-22 13:39:14.618 +0000] INFO  [pipeline-thread-18] - c.c.l.p.s.PipelineRepositoryService: Pipeline '5d013da5-b8dc-4900-85e1-f3f9845605b9/child-00000-ddbe1718-937b-4302-849f-a94fe8ecd3d9': RUNNING -> ERROR
[2017-02-22 13:39:14.641 +0000] ERROR [pipeline-thread-19] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Problem allocating instances.
        at com.cloudera.launchpad.pluggable.common.ExceptionConditions.propagateWithSuppressedDetails(ExceptionConditions.java:114)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:534)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:480)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:458)
        at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:306)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:154)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:134)
        at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
        at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
        at sun.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
        at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$c8e98257.runUnchecked(<generated>)

The rest of the logs were pretty similar but that was the main part of it. I see the error that it's running out of disk space but where? Is it in the manager instance because the disk size on that instance is the default value of 60 gb. Is is because my worker/gateway/master instances are 'n1-standard-1'? Some help would be appreciated.

0 REPLIES 0