Cloudbreak fails to create my cluster on AWS

[#1]

When I tried to create a cluster on AWS I get the following exceptions...

cloudbreak_1 | 2018-02-01 21:00:12,797 [http-nio-8080-exec-6] create:186 INFO c.s.c.s.c.AmbariClusterService - [owner:a290539e-7056-4492-8831-23d497654084] [type:springLog] [id:] [name:] Cluster requested [BlueprintId: 1]
cloudbreak_1 | 2018-02-01 21:00:13,011 [reactorDispatcher-9] validateImageOptIn:164 ERROR c.s.c.c.a.AwsSetup - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] Image opt-in could not be validated for AMI 'ami-7cb35904'.
cloudbreak_1 | com.amazonaws.AmazonClientException: Unrecognized service response for the dry-run request.
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.dryRun(AmazonEC2Client.java:13557)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.aws.AwsSetup.validateImageOptIn(AwsSetup.java:148)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.aws.AwsSetup.prerequisites(AwsSetup.java:108)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:50)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:21)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$FastClassBySpringCGLIB$$bbcf5b79.invoke(<generated>)
cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$EnhancerBySpringCGLIB$$ed30f4ac.accept(<generated>)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
cloudbreak_1 | at java.lang.Thread.run(Thread.java:745)
cloudbreak_1 | Caused by: com.amazonaws.services.ec2.model.AmazonEC2Exception: No default VPC for this user (Service: AmazonEC2; Status Code: 400; Error Code: VPCIdNotSpecified; Request ID: e0fbafe7-f44e-4bc0-be32-5e31eda2b17a)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:13611)
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:13587)
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.dryRun(AmazonEC2Client.java:13549)
cloudbreak_1 | ... 25 common frames omitted
cloudbreak_1 | 2018-02-01 21:00:13,056 [reactorDispatcher-9] accept:140 DEBUG c.s.c.c.f.Flow2Handler - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] flow control event arrived: key: SETUPRESULT_ERROR, flowid: 384a1d99-4eba-4e16-ba29-5e71534c852a, payload: CloudPlatformResult{status=FAILED, statusReason='You are not authorized to perform this operation.', errorDetails=com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: You are not authorized to perform this operation., request=CloudStackRequest{, cloudStack=CloudStack{groups=[com.sequenceiq.cloudbreak.cloud.model.Group@c94202e, com.sequenceiq.cloudbreak.cloud.model.Group@6491d02d, com.sequenceiq.cloudbreak.cloud.model.Group@43807e4d, com.sequenceiq.cloudbreak.cloud.model.Group@4f5952db, com.sequenceiq.cloudbreak.cloud.model.Group@62e19e4e], network=com.sequenceiq.cloudbreak.cloud.model.Network@81b3a87, image=Image{imageName='ami-7cb35904', userdata={CORE=#!/bin/bash
cloudbreak_1 |
cloudbreak_1 | ## logging
cloudbreak_1 | exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
cloudbreak_1 |
cloudbreak_1 | set -x
cloudbreak_1 |
cloudbreak_1 | export CLOUD_PLATFORM="AWS"
cloudbreak_1 | export START_LABEL=97
cloudbreak_1 | export PLATFORM_DISK_PREFIX=xvd
cloudbreak_1 | export LAZY_FORMAT_DISK_LIMIT=12
cloudbreak_1 | export IS_GATEWAY=false
cloudbreak_1 | export TMP_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCbuDfdt6ukwFNintr+cEQGY/X0sKSnXIsW1jh1OOdJYUMM89fCgpRbF1ixUEPo1xAau6R+iRVt9fx9tavGc1nYnj3F/CxRkK2uYrlfDzBsDXTDJUqvcGddF4KgkJzof4u/nQEhQnsJE5RQWqj9lOrAxscIDXD3WysLN6x09ih+lgtg+sK1/W37RqLLA+ECEpFIUq5k/8QadS+8pCR6ITenoF0b62LSBcDod64bOuWddRtBaQnY7XkL9lGWESiPA/Mj+pQzpignmuNijnXvH+UHvH9zikK02/Ee0E6ySveE4MDAtFc+1AbIf5benDsZoJQuwD1/MCR4ZYmSBaYdX9H9 cloudbreak
cloudbreak_1 | "
cloudbreak_1 | export PUBLIC_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCRxh8AoPplqmlo0fP2F2uTyh073gKvLARehGwtetY7JOtn2SDf1h5M7q3Ilq4/Nl9KGoTRzMtXlVPixDsldImLtuwipS0+BAjjpdsXIeTSCmSYWS6DUh2PG/UP0gdX8TIFBhElCqhGnTPKhhXCmXH0GFlx/Z3rTTK6khsahEtH9R+tvBdPa70D6yLbVFW578UBb8TzfsAmbcRlPojn79CaW6/LYP8I+1dxo/7MGU31yJb2sF4YEMiEdA0Mu5k8rogyUqlJbpcAHJLW4y+elBW7A8KUHF+3RcWsLBiFUBt+22mC2TR8PaDomjlLEmO07vlF61Zh1LeIWiIVg1D9Ee4d rt2357@att.com"
cloudbreak_1 | export RELOCATE_DOCKER=false
cloudbreak_1 | export SSH_USER=cloudbreak
cloudbreak_1 | export SALT_BOOT_PASSWORD=2cq7kg6360sk0e6qrq4o1nk9dp
cloudbreak_1 | export SALT_BOOT_SIGN_KEY=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAm7g33berpMBTYp7a/nBEBmP19LCkp1yLFtY4dTjnSWFDDPPXwoKUWxdYsVBD6NcQGrukfokVbfX8fbWrxnNZ2J49xfwsUZCtrmK5Xw8wbA10wyVKr3BnXReCoJCc6H+Lv50BIUJ7CROUUFqo/ZTqwMbHCA1w91srCzesdPYofpYLYPrCtf1t+0aiywPhAhKRSFKuZP/EGnUvvKQkeiE3p6BdG+ti0gXA6HeuGzrlnXUbQWkJ2O15C/ZRlhEojwPzI/qUM6YoJ5rjYo517x/lB7x/c4pCtNvxHtBOskr3hODAwLRXPtQGyH+W3pw7GaCULsA9fzAkeGWJkgWmHV/R/QIDAQAB
cloudbreak_1 |
cloudbreak_1 | touch /tmp/cb-custom-data-default.txt
cloudbreak_1 |
cloudbreak_1 | /usr/bin/user-data-helper.sh "$@" &> /var/log/user-data.log
cloudbreak_1 | , GATEWAY=#!/bin/bash
cloudbreak_1 |
cloudbreak_1 | ## logging
cloudbreak_1 | exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
cloudbreak_1 |
cloudbreak_1 | set -x
cloudbreak_1 |
cloudbreak_1 | export CLOUD_PLATFORM="AWS"
cloudbreak_1 | export START_LABEL=97
cloudbreak_1 | export PLATFORM_DISK_PREFIX=xvd
cloudbreak_1 | export LAZY_FORMAT_DISK_LIMIT=12
cloudbreak_1 | export IS_GATEWAY=true
cloudbreak_1 | export TMP_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCbuDfdt6ukwFNintr+cEQGY/X0sKSnXIsW1jh1OOdJYUMM89fCgpRbF1ixUEPo1xAau6R+iRVt9fx9tavGc1nYnj3F/CxRkK2uYrlfDzBsDXTDJUqvcGddF4KgkJzof4u/nQEhQnsJE5RQWqj9lOrAxscIDXD3WysLN6x09ih+lgtg+sK1/W37RqLLA+ECEpFIUq5k/8QadS+8pCR6ITenoF0b62LSBcDod64bOuWddRtBaQnY7XkL9lGWESiPA/Mj+pQzpignmuNijnXvH+UHvH9zikK02/Ee0E6ySveE4MDAtFc+1AbIf5benDsZoJQuwD1/MCR4ZYmSBaYdX9H9 cloudbreak
cloudbreak_1 | "
cloudbreak_1 | export PUBLIC_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCRxh8AoPplqmlo0fP2F2uTyh073gKvLARehGwtetY7JOtn2SDf1h5M7q3Ilq4/Nl9KGoTRzMtXlVPixDsldImLtuwipS0+BAjjpdsXIeTSCmSYWS6DUh2PG/UP0gdX8TIFBhElCqhGnTPKhhXCmXH0GFlx/Z3rTTK6khsahEtH9R+tvBdPa70D6yLbVFW578UBb8TzfsAmbcRlPojn79CaW6/LYP8I+1dxo/7MGU31yJb2sF4YEMiEdA0Mu5k8rogyUqlJbpcAHJLW4y+elBW7A8KUHF+3RcWsLBiFUBt+22mC2TR8PaDomjlLEmO07vlF61Zh1LeIWiIVg1D9Ee4d rt2357@att.com"
cloudbreak_1 | export RELOCATE_DOCKER=false
cloudbreak_1 | export SSH_USER=cloudbreak
cloudbreak_1 | export SALT_BOOT_PASSWORD=2cq7kg6360sk0e6qrq4o1nk9dp
cloudbreak_1 | export SALT_BOOT_SIGN_KEY=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAm7g33berpMBTYp7a/nBEBmP19LCkp1yLFtY4dTjnSWFDDPPXwoKUWxdYsVBD6NcQGrukfokVbfX8fbWrxnNZ2J49xfwsUZCtrmK5Xw8wbA10wyVKr3BnXReCoJCc6H+Lv50BIUJ7CROUUFqo/ZTqwMbHCA1w91srCzesdPYofpYLYPrCtf1t+0aiywPhAhKRSFKuZP/EGnUvvKQkeiE3p6BdG+ti0gXA6HeuGzrlnXUbQWkJ2O15C/ZRlhEojwPzI/qUM6YoJ5rjYo517x/lB7x/c4pCtNvxHtBOskr3hODAwLRXPtQGyH+W3pw7GaCULsA9fzAkeGWJkgWmHV/R/QIDAQAB
cloudbreak_1 |
cloudbreak_1 | touch /tmp/cb-custom-data-default.txt
cloudbreak_1 |
cloudbreak_1 | /usr/bin/user-data-helper.sh "$@" &> /var/log/user-data.log
cloudbreak_1 | }}}}}
cloudbreak_1 | 2018-02-01 21:00:13,124 [reactorDispatcher-9] execute:73 INFO c.s.c.c.f.AbstractAction - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] Stack: 7, flow state: SETUP_STATE, phase: service, execution time 0 sec
cloudbreak_1 | 2018-02-01 21:00:13,125 [reactorDispatcher-9] handleStackCreationFailure:231 ERROR c.s.c.c.f.s.p.a.StackCreationService - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] Error during stack creation flow:
cloudbreak_1 | com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: You are not authorized to perform this operation.
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.aws.AwsSetup.prerequisites(AwsSetup.java:118)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:50)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:21)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$FastClassBySpringCGLIB$$bbcf5b79.invoke(<generated>)
cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$EnhancerBySpringCGLIB$$ed30f4ac.accept(<generated>)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
cloudbreak_1 | at java.lang.Thread.run(Thread.java:745)
cloudbreak_1 | 2018-02-01 21:00:13,125 [reactorDispatcher-9] fireEventAndLog:25 DEBUG c.s.c.c.f.s.FlowMessageService - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] STACK_INFRASTRUCTURE_CREATE_FAILED [STACK_FLOW_STEP].

[#2]

@Bob Thorman

I think you are using Hortonworks Data Cloud. If yes then please the instruction here follow https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.16.2/bk_hdcloud-aws/content/subscr...

[#3]

I instantiated cloudbreak-deployer-1144-2017-05-151 (ami-e70c6b87) from the AWS Community AMI's section of the AWS Marketplace. Are you saying I have to pay for another HDP service to get this to work?

[#4]

ok then you don't use hdc.

Are you saying I have to pay for another HDP service to get this to work? No

First of all please upgrade for latest GA release with 'cbd update' command.

[#5]

I already did the 'cbd update' and I'm on the latest release.

[#6]

After reading the HTX Data Cloud docs you provide, I'm sure I don't need a subscription to those since I have an agreement for CloudBreak. So how do I get around this?

[#7]

Hi @Bob Thorman,

It looks like you tried launching a cluster from an AMI that is supposed to only work with HDCloud. You must follow the instructions in the HDCloud documentation to make this work. Here is the latest doc link: https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.16.5/bk_hdcloud-aws/content/launch...

First you need to launch HDCloud and then you can use it to create clusters.

Hope this helps.

[#8 · Accepted solution]

See this answer...How do I get CloudBreak to create a cluster in AWS: image opt-in could not be validated