Cloudbreak fails to create my cluster on AWS

rt2357 — Fri, 02 Feb 2018 22:12:37 GMT

When I tried to create a cluster on AWS I get the following exceptions...

cloudbreak_1 | 2018-02-01 21:00:12,797 [http-nio-8080-exec-6] create:186 INFO c.s.c.s.c.AmbariClusterService - [owner:a290539e-7056-4492-8831-23d497654084] [type:springLog] [id:] [name:] Cluster requested [BlueprintId: 1]
cloudbreak_1 | 2018-02-01 21:00:13,011 [reactorDispatcher-9] validateImageOptIn:164 ERROR c.s.c.c.a.AwsSetup - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] Image opt-in could not be validated for AMI 'ami-7cb35904'.
cloudbreak_1 | com.amazonaws.AmazonClientException: Unrecognized service response for the dry-run request.
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.dryRun(AmazonEC2Client.java:13557)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.aws.AwsSetup.validateImageOptIn(AwsSetup.java:148)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.aws.AwsSetup.prerequisites(AwsSetup.java:108)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:50)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:21)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$FastClassBySpringCGLIB$$bbcf5b79.invoke(<generated>)
cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$EnhancerBySpringCGLIB$$ed30f4ac.accept(<generated>)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
cloudbreak_1 | at java.lang.Thread.run(Thread.java:745)
cloudbreak_1 | Caused by: com.amazonaws.services.ec2.model.AmazonEC2Exception: No default VPC for this user (Service: AmazonEC2; Status Code: 400; Error Code: VPCIdNotSpecified; Request ID: e0fbafe7-f44e-4bc0-be32-5e31eda2b17a)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:13611)
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:13587)
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.dryRun(AmazonEC2Client.java:13549)
cloudbreak_1 | ... 25 common frames omitted
cloudbreak_1 | 2018-02-01 21:00:13,056 [reactorDispatcher-9] accept:140 DEBUG c.s.c.c.f.Flow2Handler - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] flow control event arrived: key: SETUPRESULT_ERROR, flowid: 384a1d99-4eba-4e16-ba29-5e71534c852a, payload: CloudPlatformResult{status=FAILED, statusReason='You are not authorized to perform this operation.', errorDetails=com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: You are not authorized to perform this operation., request=CloudStackRequest{, cloudStack=CloudStack{groups=[com.sequenceiq.cloudbreak.cloud.model.Group@c94202e, com.sequenceiq.cloudbreak.cloud.model.Group@6491d02d, com.sequenceiq.cloudbreak.cloud.model.Group@43807e4d, com.sequenceiq.cloudbreak.cloud.model.Group@4f5952db, com.sequenceiq.cloudbreak.cloud.model.Group@62e19e4e], network=com.sequenceiq.cloudbreak.cloud.model.Network@81b3a87, image=Image{imageName='ami-7cb35904', userdata={CORE=#!/bin/bash
cloudbreak_1 |
cloudbreak_1 | ## logging
cloudbreak_1 | exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
cloudbreak_1 |
cloudbreak_1 | set -x
cloudbreak_1 |
cloudbreak_1 | export CLOUD_PLATFORM="AWS"
cloudbreak_1 | export START_LABEL=97
cloudbreak_1 | export PLATFORM_DISK_PREFIX=xvd
cloudbreak_1 | export LAZY_FORMAT_DISK_LIMIT=12
cloudbreak_1 | export IS_GATEWAY=false
cloudbreak_1 | export TMP_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCbuDfdt6ukwFNintr+cEQGY/X0sKSnXIsW1jh1OOdJYUMM89fCgpRbF1ixUEPo1xAau6R+iRVt9fx9tavGc1nYnj3F/CxRkK2uYrlfDzBsDXTDJUqvcGddF4KgkJzof4u/nQEhQnsJE5RQWqj9lOrAxscIDXD3WysLN6x09ih+lgtg+sK1/W37RqLLA+ECEpFIUq5k/8QadS+8pCR6ITenoF0b62LSBcDod64bOuWddRtBaQnY7XkL9lGWESiPA/Mj+pQzpignmuNijnXvH+UHvH9zikK02/Ee0E6ySveE4MDAtFc+1AbIf5benDsZoJQuwD1/MCR4ZYmSBaYdX9H9 cloudbreak
cloudbreak_1 | "
cloudbreak_1 | export PUBLIC_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCRxh8AoPplqmlo0fP2F2uTyh073gKvLARehGwtetY7JOtn2SDf1h5M7q3Ilq4/Nl9KGoTRzMtXlVPixDsldImLtuwipS0+BAjjpdsXIeTSCmSYWS6DUh2PG/UP0gdX8TIFBhElCqhGnTPKhhXCmXH0GFlx/Z3rTTK6khsahEtH9R+tvBdPa70D6yLbVFW578UBb8TzfsAmbcRlPojn79CaW6/LYP8I+1dxo/7MGU31yJb2sF4YEMiEdA0Mu5k8rogyUqlJbpcAHJLW4y+elBW7A8KUHF+3RcWsLBiFUBt+22mC2TR8PaDomjlLEmO07vlF61Zh1LeIWiIVg1D9Ee4d rt2357@att.com"
cloudbreak_1 | export RELOCATE_DOCKER=false
cloudbreak_1 | export SSH_USER=cloudbreak
cloudbreak_1 | export SALT_BOOT_PASSWORD=2cq7kg6360sk0e6qrq4o1nk9dp
cloudbreak_1 | export SALT_BOOT_SIGN_KEY=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAm7g33berpMBTYp7a/nBEBmP19LCkp1yLFtY4dTjnSWFDDPPXwoKUWxdYsVBD6NcQGrukfokVbfX8fbWrxnNZ2J49xfwsUZCtrmK5Xw8wbA10wyVKr3BnXReCoJCc6H+Lv50BIUJ7CROUUFqo/ZTqwMbHCA1w91srCzesdPYofpYLYPrCtf1t+0aiywPhAhKRSFKuZP/EGnUvvKQkeiE3p6BdG+ti0gXA6HeuGzrlnXUbQWkJ2O15C/ZRlhEojwPzI/qUM6YoJ5rjYo517x/lB7x/c4pCtNvxHtBOskr3hODAwLRXPtQGyH+W3pw7GaCULsA9fzAkeGWJkgWmHV/R/QIDAQAB
cloudbreak_1 |
cloudbreak_1 | touch /tmp/cb-custom-data-default.txt
cloudbreak_1 |
cloudbreak_1 | /usr/bin/user-data-helper.sh "$@" &> /var/log/user-data.log
cloudbreak_1 | , GATEWAY=#!/bin/bash
cloudbreak_1 |
cloudbreak_1 | ## logging
cloudbreak_1 | exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
cloudbreak_1 |
cloudbreak_1 | set -x
cloudbreak_1 |
cloudbreak_1 | export CLOUD_PLATFORM="AWS"
cloudbreak_1 | export START_LABEL=97
cloudbreak_1 | export PLATFORM_DISK_PREFIX=xvd
cloudbreak_1 | export LAZY_FORMAT_DISK_LIMIT=12
cloudbreak_1 | export IS_GATEWAY=true
cloudbreak_1 | export TMP_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCbuDfdt6ukwFNintr+cEQGY/X0sKSnXIsW1jh1OOdJYUMM89fCgpRbF1ixUEPo1xAau6R+iRVt9fx9tavGc1nYnj3F/CxRkK2uYrlfDzBsDXTDJUqvcGddF4KgkJzof4u/nQEhQnsJE5RQWqj9lOrAxscIDXD3WysLN6x09ih+lgtg+sK1/W37RqLLA+ECEpFIUq5k/8QadS+8pCR6ITenoF0b62LSBcDod64bOuWddRtBaQnY7XkL9lGWESiPA/Mj+pQzpignmuNijnXvH+UHvH9zikK02/Ee0E6ySveE4MDAtFc+1AbIf5benDsZoJQuwD1/MCR4ZYmSBaYdX9H9 cloudbreak
cloudbreak_1 | "
cloudbreak_1 | export PUBLIC_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCRxh8AoPplqmlo0fP2F2uTyh073gKvLARehGwtetY7JOtn2SDf1h5M7q3Ilq4/Nl9KGoTRzMtXlVPixDsldImLtuwipS0+BAjjpdsXIeTSCmSYWS6DUh2PG/UP0gdX8TIFBhElCqhGnTPKhhXCmXH0GFlx/Z3rTTK6khsahEtH9R+tvBdPa70D6yLbVFW578UBb8TzfsAmbcRlPojn79CaW6/LYP8I+1dxo/7MGU31yJb2sF4YEMiEdA0Mu5k8rogyUqlJbpcAHJLW4y+elBW7A8KUHF+3RcWsLBiFUBt+22mC2TR8PaDomjlLEmO07vlF61Zh1LeIWiIVg1D9Ee4d rt2357@att.com"
cloudbreak_1 | export RELOCATE_DOCKER=false
cloudbreak_1 | export SSH_USER=cloudbreak
cloudbreak_1 | export SALT_BOOT_PASSWORD=2cq7kg6360sk0e6qrq4o1nk9dp
cloudbreak_1 | export SALT_BOOT_SIGN_KEY=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAm7g33berpMBTYp7a/nBEBmP19LCkp1yLFtY4dTjnSWFDDPPXwoKUWxdYsVBD6NcQGrukfokVbfX8fbWrxnNZ2J49xfwsUZCtrmK5Xw8wbA10wyVKr3BnXReCoJCc6H+Lv50BIUJ7CROUUFqo/ZTqwMbHCA1w91srCzesdPYofpYLYPrCtf1t+0aiywPhAhKRSFKuZP/EGnUvvKQkeiE3p6BdG+ti0gXA6HeuGzrlnXUbQWkJ2O15C/ZRlhEojwPzI/qUM6YoJ5rjYo517x/lB7x/c4pCtNvxHtBOskr3hODAwLRXPtQGyH+W3pw7GaCULsA9fzAkeGWJkgWmHV/R/QIDAQAB
cloudbreak_1 |
cloudbreak_1 | touch /tmp/cb-custom-data-default.txt
cloudbreak_1 |
cloudbreak_1 | /usr/bin/user-data-helper.sh "$@" &> /var/log/user-data.log
cloudbreak_1 | }}}}}
cloudbreak_1 | 2018-02-01 21:00:13,124 [reactorDispatcher-9] execute:73 INFO c.s.c.c.f.AbstractAction - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] Stack: 7, flow state: SETUP_STATE, phase: service, execution time 0 sec
cloudbreak_1 | 2018-02-01 21:00:13,125 [reactorDispatcher-9] handleStackCreationFailure:231 ERROR c.s.c.c.f.s.p.a.StackCreationService - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] Error during stack creation flow:
cloudbreak_1 | com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: You are not authorized to perform this operation.
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.aws.AwsSetup.prerequisites(AwsSetup.java:118)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:50)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:21)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$FastClassBySpringCGLIB$$bbcf5b79.invoke(<generated>)
cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$EnhancerBySpringCGLIB$$ed30f4ac.accept(<generated>)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
cloudbreak_1 | at java.lang.Thread.run(Thread.java:745)
cloudbreak_1 | 2018-02-01 21:00:13,125 [reactorDispatcher-9] fireEventAndLog:25 DEBUG c.s.c.c.f.s.FlowMessageService - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] STACK_INFRASTRUCTURE_CREATE_FAILED [STACK_FLOW_STEP].

Re: Cloudbreak fails to create my cluster on AWS

rdoktorics — Sat, 03 Feb 2018 00:01:10 GMT

@Bob Thorman

I think you are using Hortonworks Data Cloud. If yes then please the instruction here follow https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.16.2/bk_hdcloud-aws/content/subscribe/index.html

Re: Cloudbreak fails to create my cluster on AWS

rt2357 — Sat, 03 Feb 2018 00:49:55 GMT

I instantiated cloudbreak-deployer-1144-2017-05-151 (ami-e70c6b87) from the AWS Community AMI's section of the AWS Marketplace. Are you saying I have to pay for another HDP service to get this to work?

Re: Cloudbreak fails to create my cluster on AWS

rdoktorics — Sat, 03 Feb 2018 01:31:40 GMT

ok then you don't use hdc.

Are you saying I have to pay for another HDP service to get this to work? No

First of all please upgrade for latest GA release with 'cbd update' command.

Re: Cloudbreak fails to create my cluster on AWS

rt2357 — Sat, 03 Feb 2018 01:40:25 GMT

After reading the HTX Data Cloud docs you provide, I'm sure I don't need a subscription to those since I have an agreement for CloudBreak. So how do I get around this?

Re: Cloudbreak fails to create my cluster on AWS

rt2357 — Sat, 03 Feb 2018 01:41:19 GMT

I already did the 'cbd update' and I'm on the latest release.

Re: Cloudbreak fails to create my cluster on AWS

Dominika — Sat, 03 Feb 2018 02:24:50 GMT

Hi @Bob Thorman,

It looks like you tried launching a cluster from an AMI that is supposed to only work with HDCloud. You must follow the instructions in the HDCloud documentation to make this work. Here is the latest doc link: https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.16.5/bk_hdcloud-aws/content/launch/index.html

First you need to launch HDCloud and then you can use it to create clusters.

Hope this helps.

Re: Cloudbreak fails to create my cluster on AWS

rt2357 — Wed, 14 Feb 2018 21:37:42 GMT

See this answer...How do I get CloudBreak to create a cluster in AWS: image opt-in could not be validated

question Re: Cloudbreak fails to create my cluster on AWS in Support Questions

Cloudbreak fails to create my cluster on AWS

Re: Cloudbreak fails to create my cluster on AWS

Re: Cloudbreak fails to create my cluster on AWS

Re: Cloudbreak fails to create my cluster on AWS

Re: Cloudbreak fails to create my cluster on AWS

Re: Cloudbreak fails to create my cluster on AWS

Re: Cloudbreak fails to create my cluster on AWS

Re: Cloudbreak fails to create my cluster on AWS