Reply
New Contributor
Posts: 2
Registered: ‎01-26-2016

Failed Director Install on AWS GovCloud

Problem:  Trying to create a CDH cluster on AWS GovCloud, but fails to build a cloudera manager instance due to a AWS "Exception while trying to allocate instance.  Specifying tags during instance launch is not supported in this region."   Is this something I can fix, or is this a problem of using Director on GovCloud.

 

Build Cloudera Director Server on Centos7 - successful

Build Cloudera Director Environment - successful

Build Cloudera Manager - failed

Build Cloudera Cluster - failed

 

Background:  Successfully built CDH on AWS cluster in non-GovCloud AWS, so I have experience successfully building a cluster.  I have only succeeded in building the Cloudera Director server in AWS GovCloud using centos 7.3 with OracleJDK 8 and the system is fully updated.

 

Full Error Log:

/var/log/cloudera-director-server/application.log

 

 


[2017-08-08 22:56:35.973 +0000] INFO [task-thread-7] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2017-08-08 22:56:35.977 +0000] INFO [task-thread-7] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2017-08-08 22:57:20.223 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.l.p.c.PluggableComputeDeploymentTemplateValidator: Validating Cloudera Manager virtual instance template
[2017-08-08 22:57:20.223 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.l.p.c.PluggableComputeInstanceTemplateValidator: Validating instance template for compute provider: aws
[2017-08-08 22:57:20.306 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.director.aws.ec2.EC2Provider: Found EC2 key name EA_GC_General_Use_Key_01 for fingerprint
[2017-08-08 22:57:20.306 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing AMI 'ami-0062d961'
[2017-08-08 22:57:20.387 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing subnet 'subnet-d2fd8aa5'
[2017-08-08 22:57:20.542 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing security group 'sg-8c930ce9'
[2017-08-08 22:57:20.642 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.d.a.e.EC2InstanceTemplateConfigurationValidator: >> Describing key pair
[2017-08-08 22:57:20.670 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating deployment template: Manager1
[2017-08-08 22:57:20.670 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating parcel repository URLs
[2017-08-08 22:57:20.670 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating CSD URLs
[2017-08-08 22:57:20.670 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating external databases and templates
[2017-08-08 22:57:20.670 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.l.m.r.v.UsageCollectionHeartbeatDeploymentTemplateValidator: Validating deployment template: Manager1
[2017-08-08 22:57:20.682 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.l.p.DatabasePipelineService: Starting pipeline '9ac90134-b857-4452-9a76-9e9efc0c4f99' with root job com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob and listener com.cloudera.launchpad.api.listeners.pipeline.BootstrapDeploymentListener
[2017-08-08 22:57:20.695 +0000] INFO [qtp2008547236-17] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.l.p.DatabasePipelineService: Create new runner thread for pipeline '9ac90134-b857-4452-9a76-9e9efc0c4f99'
[2017-08-08 22:57:20.723 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments - - c.c.l.d.DeploymentRepositoryService: Deployment 'Manager1': BOOTSTRAPPING -> BOOTSTRAPPING
[2017-08-08 22:57:20.766 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob - c.c.l.pipeline.util.PipelineRunner: >> DefaultBootstrapDeploymentJob/1 [CreateDeploymentContext{environment=Environment{name='Environment1', provider=InstanceProviderConfi ...
[2017-08-08 22:57:20.912 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=62, pipeline=9ac90134-b857-4452-9a76-9e9efc0c4f99, c ...
[2017-08-08 22:57:20.966 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Requesting an instance for Cloudera Manager]
[2017-08-08 22:57:20.973 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.launchpad.pipeline.AbstractJob: Requesting an instance for Cloudera Manager
[2017-08-08 22:57:20.973 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.pipeline.SetStatusJob - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-08-08 22:57:21.005 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances/2 [VirtualInstanceGroup{name='CM', virtualInstances=[VirtualInstance{id='80ab0ad0-65ff-4ce8-8780-6580d ...
[2017-08-08 22:57:21.048 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=73, pipeline=9ac90134-b857-4452-9a76-9e9efc0c4f99, c ...
[2017-08-08 22:57:21.077 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: >> AllocateInstances$AllocateAndWaitForInstancesToRun/2 [VirtualInstanceGroup{name='CM', virtualInstances=[VirtualInstance{id='80ab0ad0-65ff-4ce8-8780-6580d ...
[2017-08-08 22:57:21.077 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Allocating 1 instances (min count 1) in group CM
[2017-08-08 22:57:21.112 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Found EC2 key name EA_GC_General_Use_Key_01 for fingerprint
[2017-08-08 22:57:21.112 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Requesting 1 instances for com.cloudera.director.aws.ec2.EC2InstanceTemplate@2dccef34
[2017-08-08 22:57:21.183 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Building 1 instance requests
[2017-08-08 22:57:21.183 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Network interface specification: {DeviceIndex: 0,SubnetId: subnet-d2fd8aa5,Groups: [sg-8c930ce9],DeleteOnTermination: true,PrivateIpAddresses: [],AssociatePublicIpAddress: true,Ipv6Addresses: [],}
[2017-08-08 22:57:21.206 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Original image block device mappings: [{DeviceName: /dev/sda1,Ebs: {SnapshotId: snap-077e44290414ed598,VolumeSize: 20,DeleteOnTermination: true,VolumeType: gp2,Encrypted: false},}]
[2017-08-08 22:57:21.206 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Block device mappings: [{DeviceName: /dev/sda1,Ebs: {SnapshotId: snap-077e44290414ed598,VolumeSize: 50,DeleteOnTermination: true,VolumeType: gp2,},}]
[2017-08-08 22:57:21.206 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Instance request type: m4.large, image: ami-0062d961
[2017-08-08 22:57:21.206 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: >> Submitted 1 run instance requests.
[2017-08-08 22:57:21.315 +0000] ERROR [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Exception while trying to allocate instance.
java.util.concurrent.ExecutionException: com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.model.AmazonEC2Exception: Specifying tags during instance launch is not supported in this region. (Service: AmazonEC2; Status Code: 400; Error Code: UnsupportedOperation; Request ID: 2047ce3a-a808-43b6-b9ca-b75020abffea)
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.cloudera.director.aws.ec2.EC2Provider.allocateOnDemandInstances(EC2Provider.java:1110)
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:639)
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:1)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:572)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:522)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:497)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:329)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:209)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:189)
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$8eac57a.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:164)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.model.AmazonEC2Exception: Specifying tags during instance launch is not supported in this region. (Service: AmazonEC2; Status Code: 400; Error Code: UnsupportedOperation; Request ID: 2047ce3a-a808-43b6-b9ca-b75020abffea)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.cloudera.director.aws.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:13611)
at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:13587)
at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2Client.executeRunInstances(AmazonEC2Client.java:13091)
at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2AsyncClient$221.call(AmazonEC2AsyncClient.java:8875)
at com.cloudera.director.aws.shaded.com.amazonaws.services.ec2.AmazonEC2AsyncClient$221.call(AmazonEC2AsyncClient.java:8869)
... 4 common frames omitted
[2017-08-08 22:57:21.316 +0000] ERROR [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Unsuccessful allocation of on demand instances. Terminating instances.
[2017-08-08 22:57:21.361 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.director.aws.ec2.EC2Provider: Unable to terminate instances, all unknown [80ab0ad0-65ff-4ce8-8780-6580db7ce48e]
[2017-08-08 22:57:21.362 +0000] ERROR [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Unexpected problem during instance allocation
at com.cloudera.launchpad.pluggable.common.ExceptionConditions.propagateWithSuppressedDetails(ExceptionConditions.java:125)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:575)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:522)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:497)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:329)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:209)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:189)
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$8eac57a.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:164)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:

Caused by: com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException: Unexpected problem during instance allocation
at com.cloudera.director.aws.ec2.EC2Provider.allocateOnDemandInstances(EC2Provider.java:1215)
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:639)
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:1)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:572)
... 36 common frames omitted
Caused by: com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException: Allocated 0 instances when the minimum count is 1.
at com.cloudera.director.aws.ec2.EC2Provider.allocateOnDemandInstances(EC2Provider.java:1174)
... 39 common frames omitted
[2017-08-08 22:57:21.362 +0000] ERROR [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: (Suppressed)
com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:

[2017-08-08 22:57:21.379 +0000] ERROR [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error ErrorInfo{code=PROVIDER_EXCEPTION, properties={message=Unexpected problem during instance allocation}, causes=[]} in job com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Unexpected problem during instance allocation
at com.cloudera.launchpad.pluggable.common.ExceptionConditions.propagateWithSuppressedDetails(ExceptionConditions.java:125)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:575)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:522)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:497)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:329)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:209)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:189)
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$8eac57a.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:164)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:

Caused by: com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException: Unexpected problem during instance allocation
at com.cloudera.director.aws.ec2.EC2Provider.allocateOnDemandInstances(EC2Provider.java:1215)
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:639)
at com.cloudera.director.aws.ec2.EC2Provider.allocate(EC2Provider.java:1)
at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:572)
... 36 common frames omitted
Caused by: com.cloudera.director.spi.v1.model.exception.UnrecoverableProviderException: Allocated 0 instances when the minimum count is 1.
at com.cloudera.director.aws.ec2.EC2Provider.allocateOnDemandInstances(EC2Provider.java:1174)
... 39 common frames omitted
[2017-08-08 22:57:21.380 +0000] ERROR [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineRunner: (Suppressed)
com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:

[2017-08-08 22:57:21.380 +0000] ERROR [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineRunner: Pipeline '9ac90134-b857-4452-9a76-9e9efc0c4f99' failed
at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$8eac57a
at com.cloudera.launchpad.bootstrap.AllocateInstances:0

[2017-08-08 22:57:21.394 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.s.PipelineRepositoryService: Pipeline '9ac90134-b857-4452-9a76-9e9efc0c4f99': RUNNING -> ERROR
[2017-08-08 22:57:21.421 +0000] INFO [p-9e9efc0c4f99-DefaultBootstrapDeploymentJob] 1ed95091-f3dd-4e54-a7e6-75d46576be3f POST /api/v9/environments/Environment1/deployments com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.d.DeploymentRepositoryService: Deployment 'Manager1': BOOTSTRAPPING -> BOOTSTRAP_FAILED
[2017-08-08 22:57:25.988 +0000] ERROR [qtp2008547236-17] 28f36d5c-d496-44bb-b974-76129e4aa9f3 GET /api/v9/environments/Environment1/deployments/Manager1/status - - c.c.l.api.v9.model.error.ErrorInfoV9: Could not translate internal errorCode: PROVIDER_EXCEPTION
[2017-08-08 22:57:30.988 +0000] ERROR [qtp2008547236-16] 6dc52b0f-689c-410f-abc5-dacb6b7e1511 GET /api/v9/environments/Environment1/deployments/Manager1/status - - c.c.l.api.v9.model.error.ErrorInfoV9: Could not translate internal errorCode: PROVIDER_EXCEPTION
[2017-08-08 22:57:35.994 +0000] ERROR [qtp2008547236-65] 42022695-e367-4612-98e1-e63ec732258c GET /api/v9/environments/Environment1/deployments/Manager1/status - - c.c.l.api.v9.model.error.ErrorInfoV9: Could not translate internal errorCode: PROVIDER_EXCEPTION
[2017-08-08 22:57:40.989 +0000] ERROR [qtp2008547236-18] e7e1d507-924c-434c-af25-6de4cf31312a GET /api/v9/environments/Environment1/deployments/Manager1/status - - c.c.l.api.v9.model.error.ErrorInfoV9: Could not translate internal errorCode: PROVIDER_EXCEPTION
[2017-08-08 22:57:45.991 +0000] ERROR [qtp2008547236-75] beadf03c-4959-4366-aa56-1a7a5201da8f GET /api/v9/environments/Environment1/deployments/Manager1/status - - c.c.l.api.v9.model.error.ErrorInfoV9: Could not translate internal errorCode: PROVIDER_EXCEPTION

 

 

 

 

 

 

 

Highlighted
Cloudera Employee
Posts: 43
Registered: ‎02-18-2014

Re: Failed Director Install on AWS GovCloud

Hello westcoastadmin,

 

Thank you for reporting this! It does appear that both the GovCloud and China AWS regions do not support tagging EC2 instances on launch, which is how Cloudera Director 2.5 performs instance tagging. For now, you will need to use Cloudera Director 2.4 or earlier, which tags after instance launch, to avoid the problem. We're looking into how to fix the problem.

 

Note that we have recently had reports that the visibility of tags for EC2 instances has been taking longer than usual - up to three minutes, I've seen - and that has caused bootstrap failures for Director 2.4 and earlier. I don't know if GovCloud has had the same altered eventual consistency. But I wanted to warn you that switching back to Director 2.4 may trade one tagging problem for another. We are also working on internal changes that should avoid this eventual consistency problem in a more comprehensive fashion, and knowing that GovCloud does not support tagging on creation will help us get things right.

Announcements