Created 02-02-2018 02:12 PM
When trying to create a cluster on AWS, I get the following exceptions;
cloudbreak_1 | 2018-02-01 21:00:12,797 [http-nio-8080-exec-6] create:186 INFO c.s.c.s.c.AmbariClusterService - [owner:a290539e-7056-4492-8831-23d497654084] [type:springLog] [id:] [name:] Cluster requested [BlueprintId: 1]
cloudbreak_1 | 2018-02-01 21:00:13,011 [reactorDispatcher-9] validateImageOptIn:164 ERROR c.s.c.c.a.AwsSetup - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] Image opt-in could not be validated for AMI 'ami-7cb35904'.
cloudbreak_1 | com.amazonaws.AmazonClientException: Unrecognized service response for the dry-run request.
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.dryRun(AmazonEC2Client.java:13557)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.aws.AwsSetup.validateImageOptIn(AwsSetup.java:148)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.aws.AwsSetup.prerequisites(AwsSetup.java:108)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:50)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:21)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$FastClassBySpringCGLIB$$bbcf5b79.invoke(<generated>)
cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$EnhancerBySpringCGLIB$$ed30f4ac.accept(<generated>)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
cloudbreak_1 | at java.lang.Thread.run(Thread.java:745)
cloudbreak_1 | Caused by: com.amazonaws.services.ec2.model.AmazonEC2Exception: No default VPC for this user (Service: AmazonEC2; Status Code: 400; Error Code: VPCIdNotSpecified; Request ID: e0fbafe7-f44e-4bc0-be32-5e31eda2b17a)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
cloudbreak_1 | at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:13611)
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:13587)
cloudbreak_1 | at com.amazonaws.services.ec2.AmazonEC2Client.dryRun(AmazonEC2Client.java:13549)
cloudbreak_1 | ... 25 common frames omitted
cloudbreak_1 | 2018-02-01 21:00:13,056 [reactorDispatcher-9] accept:140 DEBUG c.s.c.c.f.Flow2Handler - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] flow control event arrived: key: SETUPRESULT_ERROR, flowid: 384a1d99-4eba-4e16-ba29-5e71534c852a, payload: CloudPlatformResult{status=FAILED, statusReason='You are not authorized to perform this operation.', errorDetails=com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: You are not authorized to perform this operation., request=CloudStackRequest{, cloudStack=CloudStack{groups=[com.sequenceiq.cloudbreak.cloud.model.Group@c94202e, com.sequenceiq.cloudbreak.cloud.model.Group@6491d02d, com.sequenceiq.cloudbreak.cloud.model.Group@43807e4d, com.sequenceiq.cloudbreak.cloud.model.Group@4f5952db, com.sequenceiq.cloudbreak.cloud.model.Group@62e19e4e], network=com.sequenceiq.cloudbreak.cloud.model.Network@81b3a87, image=Image{imageName='ami-7cb35904', userdata={CORE=#!/bin/bash
cloudbreak_1 |
cloudbreak_1 | ## logging
cloudbreak_1 | exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
cloudbreak_1 |
cloudbreak_1 | set -x
cloudbreak_1 |
cloudbreak_1 | export CLOUD_PLATFORM="AWS"
cloudbreak_1 | export START_LABEL=97
cloudbreak_1 | export PLATFORM_DISK_PREFIX=xvd
cloudbreak_1 | export LAZY_FORMAT_DISK_LIMIT=12
cloudbreak_1 | export IS_GATEWAY=false
cloudbreak_1 | export TMP_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCbuDfdt6ukwFNintr+cEQGY/X0sKSnXIsW1jh1OOdJYUMM89fCgpRbF1ixUEPo1xAau6R+iRVt9fx9tavGc1nYnj3F/CxRkK2uYrlfDzBsDXTDJUqvcGddF4KgkJzof4u/nQEhQnsJE5RQWqj9lOrAxscIDXD3WysLN6x09ih+lgtg+sK1/W37RqLLA+ECEpFIUq5k/8QadS+8pCR6ITenoF0b62LSBcDod64bOuWddRtBaQnY7XkL9lGWESiPA/Mj+pQzpignmuNijnXvH+UHvH9zikK02/Ee0E6ySveE4MDAtFc+1AbIf5benDsZoJQuwD1/MCR4ZYmSBaYdX9H9 cloudbreak
cloudbreak_1 | "
cloudbreak_1 | export PUBLIC_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCRxh8AoPplqmlo0fP2F2uTyh073gKvLARehGwtetY7JOtn2SDf1h5M7q3Ilq4/Nl9KGoTRzMtXlVPixDsldImLtuwipS0+BAjjpdsXIeTSCmSYWS6DUh2PG/UP0gdX8TIFBhElCqhGnTPKhhXCmXH0GFlx/Z3rTTK6khsahEtH9R+tvBdPa70D6yLbVFW578UBb8TzfsAmbcRlPojn79CaW6/LYP8I+1dxo/7MGU31yJb2sF4YEMiEdA0Mu5k8rogyUqlJbpcAHJLW4y+elBW7A8KUHF+3RcWsLBiFUBt+22mC2TR8PaDomjlLEmO07vlF61Zh1LeIWiIVg1D9Ee4d rt2357@att.com"
cloudbreak_1 | export RELOCATE_DOCKER=false
cloudbreak_1 | export SSH_USER=cloudbreak
cloudbreak_1 | export SALT_BOOT_PASSWORD=2cq7kg6360sk0e6qrq4o1nk9dp
cloudbreak_1 | export SALT_BOOT_SIGN_KEY=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAm7g33berpMBTYp7a/nBEBmP19LCkp1yLFtY4dTjnSWFDDPPXwoKUWxdYsVBD6NcQGrukfokVbfX8fbWrxnNZ2J49xfwsUZCtrmK5Xw8wbA10wyVKr3BnXReCoJCc6H+Lv50BIUJ7CROUUFqo/ZTqwMbHCA1w91srCzesdPYofpYLYPrCtf1t+0aiywPhAhKRSFKuZP/EGnUvvKQkeiE3p6BdG+ti0gXA6HeuGzrlnXUbQWkJ2O15C/ZRlhEojwPzI/qUM6YoJ5rjYo517x/lB7x/c4pCtNvxHtBOskr3hODAwLRXPtQGyH+W3pw7GaCULsA9fzAkeGWJkgWmHV/R/QIDAQAB
cloudbreak_1 |
cloudbreak_1 | touch /tmp/cb-custom-data-default.txt
cloudbreak_1 |
cloudbreak_1 | /usr/bin/user-data-helper.sh "$@" &> /var/log/user-data.log
cloudbreak_1 | , GATEWAY=#!/bin/bash
cloudbreak_1 |
cloudbreak_1 | ## logging
cloudbreak_1 | exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
cloudbreak_1 |
cloudbreak_1 | set -x
cloudbreak_1 |
cloudbreak_1 | export CLOUD_PLATFORM="AWS"
cloudbreak_1 | export START_LABEL=97
cloudbreak_1 | export PLATFORM_DISK_PREFIX=xvd
cloudbreak_1 | export LAZY_FORMAT_DISK_LIMIT=12
cloudbreak_1 | export IS_GATEWAY=true
cloudbreak_1 | export TMP_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCbuDfdt6ukwFNintr+cEQGY/X0sKSnXIsW1jh1OOdJYUMM89fCgpRbF1ixUEPo1xAau6R+iRVt9fx9tavGc1nYnj3F/CxRkK2uYrlfDzBsDXTDJUqvcGddF4KgkJzof4u/nQEhQnsJE5RQWqj9lOrAxscIDXD3WysLN6x09ih+lgtg+sK1/W37RqLLA+ECEpFIUq5k/8QadS+8pCR6ITenoF0b62LSBcDod64bOuWddRtBaQnY7XkL9lGWESiPA/Mj+pQzpignmuNijnXvH+UHvH9zikK02/Ee0E6ySveE4MDAtFc+1AbIf5benDsZoJQuwD1/MCR4ZYmSBaYdX9H9 cloudbreak
cloudbreak_1 | "
cloudbreak_1 | export PUBLIC_SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCRxh8AoPplqmlo0fP2F2uTyh073gKvLARehGwtetY7JOtn2SDf1h5M7q3Ilq4/Nl9KGoTRzMtXlVPixDsldImLtuwipS0+BAjjpdsXIeTSCmSYWS6DUh2PG/UP0gdX8TIFBhElCqhGnTPKhhXCmXH0GFlx/Z3rTTK6khsahEtH9R+tvBdPa70D6yLbVFW578UBb8TzfsAmbcRlPojn79CaW6/LYP8I+1dxo/7MGU31yJb2sF4YEMiEdA0Mu5k8rogyUqlJbpcAHJLW4y+elBW7A8KUHF+3RcWsLBiFUBt+22mC2TR8PaDomjlLEmO07vlF61Zh1LeIWiIVg1D9Ee4d rt2357@att.com"
cloudbreak_1 | export RELOCATE_DOCKER=false
cloudbreak_1 | export SSH_USER=cloudbreak
cloudbreak_1 | export SALT_BOOT_PASSWORD=2cq7kg6360sk0e6qrq4o1nk9dp
cloudbreak_1 | export SALT_BOOT_SIGN_KEY=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAm7g33berpMBTYp7a/nBEBmP19LCkp1yLFtY4dTjnSWFDDPPXwoKUWxdYsVBD6NcQGrukfokVbfX8fbWrxnNZ2J49xfwsUZCtrmK5Xw8wbA10wyVKr3BnXReCoJCc6H+Lv50BIUJ7CROUUFqo/ZTqwMbHCA1w91srCzesdPYofpYLYPrCtf1t+0aiywPhAhKRSFKuZP/EGnUvvKQkeiE3p6BdG+ti0gXA6HeuGzrlnXUbQWkJ2O15C/ZRlhEojwPzI/qUM6YoJ5rjYo517x/lB7x/c4pCtNvxHtBOskr3hODAwLRXPtQGyH+W3pw7GaCULsA9fzAkeGWJkgWmHV/R/QIDAQAB
cloudbreak_1 |
cloudbreak_1 | touch /tmp/cb-custom-data-default.txt
cloudbreak_1 |
cloudbreak_1 | /usr/bin/user-data-helper.sh "$@" &> /var/log/user-data.log
cloudbreak_1 | }}}}}
cloudbreak_1 | 2018-02-01 21:00:13,124 [reactorDispatcher-9] execute:73 INFO c.s.c.c.f.AbstractAction - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] Stack: 7, flow state: SETUP_STATE, phase: service, execution time 0 sec
cloudbreak_1 | 2018-02-01 21:00:13,125 [reactorDispatcher-9] handleStackCreationFailure:231 ERROR c.s.c.c.f.s.p.a.StackCreationService - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] Error during stack creation flow:
cloudbreak_1 | com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: You are not authorized to perform this operation.
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.aws.AwsSetup.prerequisites(AwsSetup.java:118)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:50)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler.accept(ProvisionSetupHandler.java:21)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$FastClassBySpringCGLIB$$bbcf5b79.invoke(<generated>)
cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.ProvisionSetupHandler$$EnhancerBySpringCGLIB$$ed30f4ac.accept(<generated>)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
cloudbreak_1 | at java.lang.Thread.run(Thread.java:745)
cloudbreak_1 | 2018-02-01 21:00:13,125 [reactorDispatcher-9] fireEventAndLog:25 DEBUG c.s.c.c.f.s.FlowMessageService - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] STACK_INFRASTRUCTURE_CREATE_FAILED [STACK_FLOW_STEP].
Created 02-09-2018 12:48 PM
As we have already spoken, the root cause was a bug in Cloudbreak when no default VPC was present in the AWS account.
The solution is to include the following line in his Profile file and do a "cbd restart" afterwards.
export DOCKER_TAG_CLOUDBREAK=1165-patch
Created 02-05-2018 08:36 PM
According to your stack trace, you may not have the requisite permissions to perform this operation. Please check your AWS user permissions and make sure you have enough capacity to allocate the cluster you are trying to allocate.
cloudbreak_1 | 2018-02-01 21:00:13,056 [reactorDispatcher-9] accept:140 DEBUG c.s.c.c.f.Flow2Handler - [owner:a290539e-7056-4492-8831-23d497654084] [type:STACK] [id:7] [name:storm] flow control event arrived: key: SETUPRESULT_ERROR, flowid: 384a1d99-4eba-4e16-ba29-5e71534c852a, payload: CloudPlatformResult{status=FAILED, statusReason='You are not authorized to perform this operation.', errorDetails=com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: You are not authorized to perform this operation., request=CloudStackRequest{, cloudStack=CloudStack{groups=[com.sequenceiq.cloudbreak.cloud.model.Group@c94202e, com.sequenceiq.cloudbreak.cloud.model.Group@6491d02d, com.sequenceiq.cloudbreak.cloud.model.Group@43807e4d, com.sequenceiq.cloudbreak.cloud.model.Group@4f5952db, com.sequenceiq.cloudbreak.cloud.model.Group@62e19e4e],
Created 02-05-2018 11:26 PM
I'm an admin in our AWS account. I don't think it was my permissions regarding resource allocation. I can create EC2 instances of the same profile manually. I think its an issue with me using the HDP AMI. Are there some special HDP polices I needs for my role?
Created 02-06-2018 07:23 PM
No, shouldn't be any special HDP policies. Perhaps you are running up against an upper quota? Cores/RAM/disk set by either AWS or your organization?
Created 02-09-2018 12:48 PM
As we have already spoken, the root cause was a bug in Cloudbreak when no default VPC was present in the AWS account.
The solution is to include the following line in his Profile file and do a "cbd restart" afterwards.
export DOCKER_TAG_CLOUDBREAK=1165-patch
Created 02-09-2018 01:08 PM
Yes, thank you @pdarvasi that was the ticket. I appreciate all your help.