Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

cloudbreak Unable to resize a cluster for a particular component

cloudbreak Unable to resize a cluster for a particular component

New Contributor

i am unable to update the group given below error , i have synced the cluster also

Stack update failed. Reason: The autoscale group's desired instance count is not match with the instance count in the cloudbreak. Desired count: 2 <> cb instance count: 0. Upscale is not possible!

3 REPLIES 3

Re: cloudbreak Unable to resize a cluster for a particular component

Expert Contributor
@navdeep agarwal

which version of cb do you using? can please contact with our support and attach the cb logs as well ?

Thanks

R

Re: cloudbreak Unable to resize a cluster for a particular component

New Contributor

Hi,

we are using cloudbreak 2.7.2

attaching scaling logs

e19,DeleteOnTermination: true,DeviceIndex: 0,Status: attached},Description: ,Groups: [{GroupName: nodes.prod-use1-cmd.k8s.hotstar-labs.com,GroupId: sg-075cb421d6d63c566}],Ipv6Addresses: [],MacAddress: 16:71:9c:62:c1:a0,NetworkInterfaceId: eni-0a4e86c4c4d720a97,OwnerId: 084690408984,PrivateDnsName: ip-10-128-101-55.ec2.internal,PrivateIpAddress: 10.128.101.55,PrivateIpAddresses: [{Primary: true,PrivateDnsName: ip-10-128-101-55.ec2.internal,PrivateIpAddress: 10.128.101.55}],SourceDestCheck: true,Status: in-use,SubnetId: subnet-00a624c510225864d,VpcId: vpc-0097791d924b5e5c7}],RootDeviceName: /dev/sda1,RootDeviceType: ebs,SecurityGroups: [{GroupName: nodes.prod-use1-cmd.k8s.hotstar-labs.com,GroupId: sg-075cb421d6d63c566}],SourceDestCheck: true,Tags: [{Key: cluster_name,Value: prod-use1-cmd.k8s.hotstar-labs.com}, {Key: Name,Value: nodes.prod-use1-cmd.k8s.hotstar-labs.com}, {Key: Stage,Value: Prod}, {Key: aws:autoscaling:groupName,Value: nodes.prod-use1-cmd.k8s.hotstar-labs.com}, {Key: KubernetesCluster,Value: prod-use1-cmd.k8s.hotstar-labs.com}, {Key: k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup,Value: nodes}, {Key: environment,Value: prod}, {Key: Team,Value: devops}, {Key: Product,Value: Hotstar}, {Key: k8s.io/role/node,Value: 1}, {Key: CostCenter,Value: India}, {Key: Owner,Value: devops-team@hotstar.com}],VirtualizationType: hvm}],OwnerId: 084690408984,RequesterId: 940372691376,ReservationId: r-0b18a14d69b1555b3,GroupNames: []}, {Groups: [],Instances: [{AmiLaunchIndex: 0,ImageId: ami-6dfe5010,InstanceId: i-08cdf229546b90aaf,InstanceType: m5.2xlarge,KeyName: kubernetes.prod-use1-cmd.k8s.hotstar-labs.com-e2:5d:69:80:8d:b5:03:52:f1:57:5f:ea:ec:6d:d4:03,LaunchTime: Sat Jan 12 13:01:08 UTC 2019,Monitoring: {State: disabled},Placement: {AvailabilityZone: us-east-1b,GroupName: ,Tenancy: default,},PrivateDnsName: ip-10-128-49-187.ec2.internal,PrivateIpAddress: 10.128.49.187,ProductCodes: [],PublicDnsName: ,State: {Code: 16,Name: running},StateTransitionReason: ,SubnetId: subnet-096e20c4365281fd5,VpcId: vpc-0097791d924b5e5c7,Architecture: x86_64,BlockDeviceMappings: [{DeviceName: /dev/sda1,Ebs: {AttachTime: Sat Jan 12 13:01:10 UTC 2019,DeleteOnTermination: true,Status: attached,VolumeId: vol-03bce8b24ac5ef7ab}}, {DeviceName: /dev/xvdbx,Ebs: {AttachTime: Mon Jan 21 09:09:20 UTC 2019,DeleteOnTermination: false,Status: attached,VolumeId: vol-06c4ff8e1d1ddcee5}}],ClientToken: 6005a108-9b41-9327-4569-f2f13237eafd_subnet-096e20c4365281fd5_1,EbsOptimized: false,EnaSupport: true,Hypervisor: xen,IamInstanceProfile: {Arn: arn:aws:iam::084690408984:instance-profile/nodes.prod-use1-cmd.k8s.hotstar-labs.com,Id: AIPAJIN7BVJEZ5FTYKZEW},ElasticGpuAssociations: [],NetworkInterfaces: [{Attachment: {AttachTime: Sat Jan 12 13:01:08 UTC 2019,AttachmentId: eni-attach-0309cc6987caadbdd,DeleteOnTermination: true,DeviceIndex: 0,Status: attached},Description: ,Groups: [{GroupName: nodes.prod-use1-cmd.k8s.hotstar-labs.com,GroupId: sg-075cb421d6d63c566}],Ipv6Addresses: [],MacAddress: 12:a4:30:a4:bd:76,NetworkInterfaceId: eni-00cb94693e97f5774,OwnerId: 084690408984,PrivateDnsName: ip-10-128-49-187.ec2.internal,PrivateIpAddress: 10.128.49.187,PrivateIpAddresses: [{Primary: true,PrivateDnsName: ip-10-128-49-187.ec2.internal,PrivateIpAddress: 10.128.49.187}],SourceDestCheck: true,Status: in-use,SubnetId: subnet-096e20c4365281fd5,VpcId: vpc-0097791d924b5e5c7}],RootDeviceName: /dev/sda1,RootDeviceType: ebs,SecurityGroups: [{GroupName: nodes.prod-use1-cmd.k8s.hotstar-labs.com,GroupId: sg-075cb421d6d63c566}],SourceDestCheck: true,Tags: [{Key: KubernetesCluster,Value: prod-use1-cmd.k8s.hotstar-labs.com}, {Key: Stage,Value: Prod}, {Key: Team,Value: devops}, {Key: environment,Value: prod}, {Key: aws:autoscaling:groupName,Value: nodes.prod-use1-cmd.k8s.hotstar-labs.com}, {Key: Owner,Value: devops-team@hotstar.com}, {Key: CostCenter,Value: India}, {Key: cluster_name,Value: prod-use1-cmd.k8s.hotstar-labs.com}, {Key: k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup,Value: nodes}, {Key: k8s.io/role/node,Value: 1}, {Key: Product,Value: Hotstar}, {Key: Name,Value: nodes.prod-use1-cmd.k8s.hotstar-labs.com}],VirtualizationType: hvm}],OwnerId: 084690408984,RequesterId: 940372691376,ReservationId: r-0930b6bdb06eac821,GroupNames: []}, {Groups: [],Instances: [{AmiLaunchIndex: 0,ImageId: ami-0454b3cc1ad84e26b,InstanceId: i-06e894241ea862cdf,InstanceType: t2.medium,KeyName: devops-master-key,LaunchTime: Thu Oct 11 07:25:41 UTC 2018,Monitoring: {State: disabled},Placement: {AvailabilityZone: us-east-1b,GroupName: ,Tenancy: default,},PrivateDnsName: ip-172-31-57-119.ec2.internal,PrivateIpAddress: 172.31.57.119,ProductCodes: [],PublicDnsName: ec2-18-212-160-2.compute-1.amazonaws.com,PublicIpAddress: 18.212.160.2,State: {Code: 16,Name: running},StateTransitionReason: ,SubnetId: subnet-c2b472e8,VpcId: vpc-2b713c4f,Architecture: x86_64,BlockDeviceMappings: [{DeviceName: /dev/sda1,Ebs: {AttachTime: Thu Oct 11 07:25:42 UTC 2018,DeleteOnTermination: true,Status: attached,VolumeId: vol-01488a29b01cef94b}}, {DeviceName: /dev/sdf,Ebs: {AttachTime: Thu Oct 11 07:25:42 UTC 2018,DeleteOnTermination: true,Status: attached,VolumeId: vol-06b9da1e8baa1eef3}}],ClientToken: ,EbsOptimized: false,EnaSupport: true,Hypervisor: xen,ElasticGpuAssociations: [],NetworkInterfaces: [{Association: {IpOwnerId: amazon,PublicDnsName: ec2-18-212-160-2.compute-1.amazonaws.com,PublicIp: 18.212.160.2},Attachment: {AttachTime: Thu Oct 11 07:25:41 UTC 2018,AttachmentId: eni-attach-0239b1abfc28b7d70,DeleteOnTermination: true,DeviceIndex: 0,Status: attached},Description: ,Groups: [{GroupName: launch-wizard-28,GroupId: sg-0a87cc88ab677fe24}],Ipv6Addresses: [],MacAddress: 12:80:e6:f7:eb:4a,NetworkInterfaceId: eni-0b4e14689503f7a81,OwnerId: 084690408984,PrivateDnsName: ip-172-31-57-119.ec2.internal,PrivateIpAddress: 172.31.57.119,PrivateIpAddresses: [{Association: {IpOwnerId: amazon,PublicDnsName: ec2-18-212-160-2.compute-1.amazonaws.com,PublicIp: 18.212.160.2},Primary: true,PrivateDnsName: ip-172-31-57-119.ec2.internal,PrivateIpAddress: 172.31.57.119}],SourceDestCheck: true,Status: in-use,SubnetId: subnet-c2b472e8,VpcId: vpc-2b713c4f}],RootDeviceName: /dev/sda1,RootDeviceType: ebs,SecurityGroups: [{GroupName: launch-wizard-28,GroupId: sg-0a87cc88ab677fe24}],SourceDestCheck: true,StateReason: {Code: Client.UserInitiatedShutdown,Message: Client.UserInitiatedShutdown: User initiated shutdown},Tags: [],VirtualizationType: hvm}],OwnerId: 084690408984,ReservationId: r-057aed294d0c51723,GroupNames: []}, {Groups: [],Instances: [{AmiLaunchIndex: 0,ImageId: ami-048486555686d18a0,InstanceId: i-0870e2c4f25ad6c6c,InstanceType: c4.xlarge,KeyName: devops-master-key,LaunchTime: Mon Dec 17 13:41:10 UTC 2018,Monitoring: {State: enabled},Placement: {AvailabilityZone: us-east-1d,GroupName: ,Tenancy: default,},PrivateDnsName: ip-10-34-21-44.ec2.internal,PrivateIpAddress: 10.34.21.44,ProductCodes: [],PublicDnsName: ,PublicIpAddress: 54.159.28.207,State: {Code: 16,Name: running},StateTransitionReason: ,SubnetId: subnet-0722875b,VpcId: vpc-2ee40854,Architecture: x86_64,BlockDeviceMappings: [{DeviceName: /dev/xvda,Ebs: {AttachTime: Mon Dec 17 13:41:11 UTC 2018,DeleteOnTermination: true,Status: attached,VolumeId: vol-004e343517c014a65}}],ClientToken: 29959ef1-9882-f109-d706-a433811e1baa_subnet-0722875b_1,EbsOptimized: false,EnaSupport: true,Hypervisor: xen,IamInstanceProfile: {Arn: arn:aws:iam::084690408984:instance-profile/eks-hs-prod20180626143751199700000003,Id: AIPAIV37ZUJHUN77UDH7S},ElasticGpuAssociations: [],NetworkInterfaces: [{Association: {IpOwnerId: amazon,PublicDnsName: ,PublicIp: 54.159.28.207},Attachment: {AttachTime: Mon Dec 17 13:41:10 UTC 2018,AttachmentId: eni-attach-02aed3f9c256f0e2c,DeleteOnTermination: true,DeviceIndex: 0,Status: attached},Description: ,Groups: [{GroupName: terraform-eks-worker,GroupId: sg-95d6b4de}],Ipv6Addresses: [],MacAddress: 0e:c6:6d:fc:56:88,NetworkInterfaceId: eni-001f9829e37272752,OwnerId: 084690408984,PrivateIpAddress: 10.34.21.44,PrivateIpAddresses: [{Association: {IpOwnerId: amazon,PublicDnsName: ,PublicIp: 54.159.28.207},Primary: true,PrivateIpAddress: 10.34.21.44}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.18.2}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.23.67}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.24.231}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.24.69}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.19.235}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.24.137}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.23.110}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.18.172}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.28.172}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.19.55}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.21.52}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.25.117}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.18.117}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.21.220}],SourceDestCheck: true,Status: in-use,SubnetId: subnet-0722875b,VpcId: vpc-2ee40854}, {Attachment: {AttachTime: Mon Dec 31 06:34:34 UTC 2018,AttachmentId: eni-attach-0bbea42c3f5644b35,DeleteOnTermination: true,DeviceIndex: 1,Status: attached},Description: aws-K8S-i-0870e2c4f25ad6c6c,Groups: [{GroupName: terraform-eks-worker,GroupId: sg-95d6b4de}],Ipv6Addresses: [],MacAddress: 0e:50:d5:ef:f0:d6,NetworkInterfaceId: eni-0bf890e471be459d9,OwnerId: 084690408984,PrivateIpAddress: 10.34.23.14,PrivateIpAddresses: [{Primary: true,PrivateIpAddress: 10.34.23.14}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.18.97}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.16.71}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.30.71}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.25.228}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.19.208}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.29.55}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.18.20}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.31.180}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.31.245}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.30.57}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.21.222}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.27.94}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.27.156}, {Primary: false,PrivateDnsName: ,PrivateIpAddress: 10.34.28.188}],SourceDestCheck: true,Status: in-use,SubnetId: subnet-0722875b,VpcId: vpc-2ee40854}],RootDeviceName: /dev/xvda,RootDeviceType: ebs,SecurityGroups: [{GroupName: terraform-eks-worker,GroupId: sg-95d6b4de}],SourceDestCheck: true,Tags: [{Key: Team,Value: DevOps}, {Key: aws:autoscaling:groupName,Value: terraform-eks-eks-worker2018062614381469310000000a}, {Key: Name,Value: terraform-eks-worker}, {Key: kubernetes.io/cluster/hs-prod,Value: owned}, {Key: CostCenter,Value: India}, {Key: Product,Value: Hotstar}, {Key: Owner,Value: devops-team@hotstar.com}],VirtualizationType: hvm}],OwnerId: 084690408984,RequesterId: 940372691376,ReservationId: r-07097a380d27ebf88,GroupNames: []}, {Groups: [],Instances: [{AmiLaunchIndex: 0,ImageId: ami-8cbbdc9a,InstanceId: i-05ada5f59a6c970df,InstanceType: c5.xlarge,KeyName: prod-dp-emr-nv,LaunchTime: Fri Oct 26 05:12:11 UTC 2018,Monitoring: {State: disabled},Placement: {AvailabilityZone: us-east-1c,GroupName: ,Tenancy: default,},PrivateDnsName: ip-10-20-0-139.ec2.internal,PrivateIpAddress: 10.20.0.139,ProductCodes: [],PublicDnsName: ,PublicIpAddress: 54.162.191.224,State: {Code: 16,Name: running},StateTransitionReason: ,SubnetId: subnet-bebea5f6,VpcId: vpc-72cff70b,Architecture: x86_64,BlockDeviceMappings: [{DeviceName: /dev/xvda,Ebs: {AttachTime: Fri Oct 26 05:12:12 UTC 2018,DeleteOnTermination: true,Status: attached,VolumeId: vol-0954659121025e6be}}],ClientToken: ,EbsOptimized: true,EnaSupport: true,Hypervisor: xen,IamInstanceProfile: {Arn: arn:aws:iam::084690408984:instance-profile/DataPipelineDefaultResourceRole,Id: AIPAJOIOP2PJSGQZZOYVA},ElasticGpuAssociations: [],NetworkInterfaces: [{Association: {IpOwnerId: amazon,PublicDnsName: ,PublicIp: 54.162.191.224},Attachment: {AttachTime: Fri Oct 26 05:12:11 UTC 2018,AttachmentId: eni-attach-08bb5f0b527c2707a,DeleteOnTermination: true,DeviceIndex: 0,Status: attached},Description: Primary network interface,Groups: [{GroupName: ads-reporting-taskrunner-us-east-1-prod,GroupId: sg-03f8fff54336c6590}],Ipv6Addresses: [],MacAddress: 0a:16:83:71:91:22,NetworkInterfaceId: eni-066c2f059c9426004,OwnerId: 084690408984,PrivateIpAddress: 10.20.0.139,PrivateIpAddresses: [{Association: {IpOwnerId: amazon,PublicDnsName: ,PublicIp: 54.162.191.224},Primary: true,PrivateIpAddress: 10.20.0.139}],SourceDestCheck: true,Status: in-use,SubnetId: subnet-bebea5f6,VpcId: vpc-72cff70b}],RootDeviceName: /dev/xvda,RootDeviceType: ebs,SecurityGroups: [{GroupName: ads-reporting-taskrunner-us-east-1-prod,GroupId: sg-03f8fff54336c6590}],SourceDestCheck: true,Tags: [{Key: CostCenter,Value: India}, {Key: Product,Value: Hotstar}, {Key: Name,Value: ads-reporting-taskrunner-us-east-1-prod}, {Key: Team,Value: AdTech}, {Key: Owner,Value: ashutosh@hotstar.com}, {Key: Stage,Value: Prod}],VirtualizationType: hvm}],OwnerId: 084690408984,ReservationId: r-0ef72624ec95c1513,GroupNames: []}],}
periscope_1    | 2019-01-30 07:12:00,024 [getThreadPoolExecutorFactoryBean-28] getRawResource:84 INFO  c.s.a.c.AmbariClientUtils - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [id:2] [cb-stack-id:2] AmbariClient getRawResource, resourceRequestMap: {path=https://10.10.242.53:9443/api/v1/clusters, query={fields=Clusters}}
uluwatu_1      | 2019-01-07T08:45:30.214Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: 7fce1681-1016-415e-a84c-d381aa9268e9] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.318 ms [-]
identity_1     | [2019-01-20 20:20:44.236] cloudfoundry-identity-server - ???? [http-nio-8080-exec-396] .... DEBUG --- UaaRequestMatcher: [loginAuthorizeRequestMatcherOld] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={login={} and headers {accept=[application/json]}
cloudbreak_1   | 2019-01-30 12:28:24,447 [reactorDispatcher-73] scalingPrerequisites:248 ERROR c.s.c.c.a.AwsSetup - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACK] [id:2] [name:hbaseha] [flow:cee721f4-1a72-407b-a1f9-984dd4aa99f5] [tracking:] The autoscale group's desired instance count is not match with the instance count in the cloudbreak. Desired count: 2 <> cb instance count: 0. Upscale is not possible!
periscope_1    | 2019-01-30 07:12:00,030 [getThreadPoolExecutorFactoryBean-2] logThreadPoolExecutorParameters:13 INFO  c.s.p.m.AbstractMonitor - [owner:periscope] [id:] [cb-stack-id:] threadpool tasks: monitor class is: com.sequenceiq.periscope.monitor.evaluator.AmbariAgentHealthEvaluator, active threads: 3, poolsize: 50, queueSize: 0, completed tasks 2692308
uluwatu_1      | 2019-01-07T08:46:30.273Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: 248e077b-fb8e-480a-98b2-f4a67b03ff44] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.477 ms [-]
identity_1     | [2019-01-20 20:20:44.237] cloudfoundry-identity-server - ???? [http-nio-8080-exec-396] .... DEBUG --- UaaRequestMatcher: [passcodeTokenMatcher] Checking match of request : '/check_token'; '/oauth/token' with parameters={grant_type=password, passcode=} and headers {accept=[application/json, application/x-www-form-urlencoded]}
cloudbreak_1   | 2019-01-30 12:28:24,448 [reactorDispatcher-73] buildLogContextForReactorHandler:69 INFO  c.s.c.l.LogContextAspects - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACK] [id:2] [name:hbaseha] [flow:cee721f4-1a72-407b-a1f9-984dd4aa99f5] [tracking:] A Reactor event handler's 'accept' method has been intercepted: execution(Flow2Handler.accept(..)), MDC logger context is built.
periscope_1    | 2019-01-30 07:12:00,031 [getThreadPoolExecutorFactoryBean-2] execute:50 INFO  c.s.p.m.AbstractMonitor - [owner:periscope] [id:] [cb-stack-id:] Succesfully submitted com.sequenceiq.periscope.monitor.evaluator.AmbariAgentHealthEvaluator for cluster 2.
uluwatu_1      | 2019-01-07T08:46:30.294Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: c8c6dbba-c815-4e5e-9430-e5dc69ce4fdf] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.333 ms [-]
identity_1     | [2019-01-20 20:20:44.237] cloudfoundry-identity-server - ???? [http-nio-8080-exec-396] .... DEBUG --- UaaRequestMatcher: [oauthAuthorizeRequestMatcher] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={response_type=token, source=credentials} and headers {accept=[application/json, application/x-www-form-urlencoded]}
cloudbreak_1   | 2019-01-30 12:28:24,471 [reactorDispatcher-73] execute:87 INFO  c.s.c.c.f.AbstractAction - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACKVIEW] [id:2] [name:hbaseha] [flow:cee721f4-1a72-407b-a1f9-984dd4aa99f5] [tracking:] Stack: 2, flow state: UPSCALE_PREVALIDATION_STATE, phase: doExec, execution time 3 sec
periscope_1    | 2019-01-30 07:12:00,041 [getThreadPoolExecutorFactoryBean-46] execute:87 INFO  c.s.p.m.e.AmbariAgentHealthEvaluator - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [id:2] [cb-stack-id:2] Checking 'Ambari Agent Heartbeat' alerts for cluster 2.
uluwatu_1      | 2019-01-07T08:47:30.359Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: 8ca18c27-61c7-441c-a4e3-681cba02c0dc] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.595 ms [-]
--
--
cloudbreak_1   | 2019-01-30 12:28:24,471 [reactorDispatcher-73] execute:87 INFO  c.s.c.c.f.AbstractAction - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACKVIEW] [id:2] [name:hbaseha] [flow:cee721f4-1a72-407b-a1f9-984dd4aa99f5] [tracking:] Stack: 2, flow state: UPSCALE_PREVALIDATION_STATE, phase: doExec, execution time 3 sec
periscope_1    | 2019-01-30 07:12:00,041 [getThreadPoolExecutorFactoryBean-46] execute:87 INFO  c.s.p.m.e.AmbariAgentHealthEvaluator - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [id:2] [cb-stack-id:2] Checking 'Ambari Agent Heartbeat' alerts for cluster 2.
uluwatu_1      | 2019-01-07T08:47:30.359Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: 8ca18c27-61c7-441c-a4e3-681cba02c0dc] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.595 ms [-]
periscope_1    | 2019-01-30 07:12:00,041 [getThreadPoolExecutorFactoryBean-46] createAmbariClient:44 INFO  c.s.p.s.AmbariClientProvider - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [id:2] [cb-stack-id:2] Create Ambari client to connect to 10.10.242.53:9443
identity_1     | [2019-01-20 20:20:44.237] cloudfoundry-identity-server - ???? [http-nio-8080-exec-396] .... DEBUG --- UaaRequestMatcher: [oauthAuthorizeApiRequestMatcher] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={response_type=code, client_id=} and headers {Authorization=[bearer ]}
periscope_1    | 2019-01-30 07:12:00,043 [getThreadPoolExecutorFactoryBean-46] getRawResource:84 INFO  c.s.a.c.AmbariClientUtils - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [id:2] [cb-stack-id:2] AmbariClient getRawResource, resourceRequestMap: {path=https://10.10.242.53:9443/api/v1/clusters, query={fields=Clusters}}
cloudbreak_1   | 2019-01-30 12:28:24,472 [reactorDispatcher-73] handleStackUpscaleFailure:168 ERROR c.s.c.c.f.s.u.StackUpscaleService - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACKVIEW] [id:2] [name:hbaseha] [flow:cee721f4-1a72-407b-a1f9-984dd4aa99f5] [tracking:] Exception during the upscale of stack
identity_1     | [2019-01-20 20:20:44.237] cloudfoundry-identity-server - ???? [http-nio-8080-exec-396] .... DEBUG --- UaaRequestMatcher: [xOauthCallbackRequestMatcher] Checking match of request : '/check_token'; '/login/callback' with parameters={code=} and headers {}
uluwatu_1      | 2019-01-07T08:47:30.383Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: c87631bd-e52c-4b72-8331-1c7034ab48d5] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.478 ms [-]
periscope_1    | 2019-01-30 07:12:00,045 [getThreadPoolExecutorFactoryBean-15] getRawResource:84 INFO  c.s.a.c.AmbariClientUtils - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [id:7] [cb-stack-id:6] AmbariClient getRawResource, resourceRequestMap: {path=https://10.10.243.96:9443/api/v1/clusters/ad-tech-user/alerts, query={fields=Alert/*}}
cloudbreak_1   | com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: The autoscale group's desired instance count is not match with the instance count in the cloudbreak. Desired count: 2 <> cb instance count: 0. Upscale is not possible!
identity_1     | [2019-01-20 20:20:44.237] cloudfoundry-identity-server - ???? [http-nio-8080-exec-396] .... DEBUG --- UaaRequestMatcher: [oauthAuthorizeRequestMatcherOld] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={response_type=token, credentials={} and headers {accept=[application/json, application/x-www-form-urlencoded]}
uluwatu_1      | 2019-01-07T08:48:30.438Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: 73e43b8b-8521-450d-bbbf-e22374de16ce] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.446 ms [-]
periscope_1    | 2019-01-30 07:12:00,059 [getThreadPoolExecutorFactoryBean-28] getRawResource:84 INFO  c.s.a.c.AmbariClientUtils - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [id:2] [cb-stack-id:2] AmbariClient getRawResource, resourceRequestMap: {path=https://10.10.242.53:9443/api/v1/clusters/hbaseha/alert_history, query={AlertHistory/definition_name=namenode_increase_in_storage_capacity_usage_weekly}}
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.aws.AwsSetup.scalingPrerequisites(AwsSetup.java:249)
identity_1     | [2019-01-20 20:20:44.239] cloudfoundry-identity-server - ???? [http-nio-8080-exec-396] .... DEBUG --- DisableIdTokenResponseTypeFilter: Processing id_token disable filter
uluwatu_1      | 2019-01-07T08:48:30.460Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: 1d4f0fde-4d9a-41b6-a0a6-4f647af74697] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.294 ms [-]
periscope_1    | 2019-01-30 07:12:00,069 [getThreadPoolExecutorFactoryBean-28] getRawResource:84 INFO  c.s.a.c.AmbariClientUtils - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [id:2] [cb-stack-id:2] AmbariClient getRawResource, resourceRequestMap: {path=https://10.10.242.53:9443/api/v1/clusters/hbaseha/alert_history/1104}
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.handler.UpscaleValidationHandler.accept(UpscaleValidationHandler.java:43)
identity_1     | [2019-01-20 20:20:44.240] cloudfoundry-identity-server - ???? [http-nio-8080-exec-396] .... DEBUG --- DisableIdTokenResponseTypeFilter: pre id_token disable:false pathinfo:null request_uri:/check_token response_type:null
uluwatu_1      | 2019-01-07T08:49:30.516Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: 7d6e592c-2f53-4934-9bfe-6d86de43be64] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.492 ms [-]
--
--
identity_1     | [2019-01-20 20:35:54.264] cloudfoundry-identity-server - ???? [http-nio-8080-exec-404] .... DEBUG --- DisableIdTokenResponseTypeFilter: Processing id_token disable filter
cloudbreak_1   | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
uluwatu_1      | 2019-01-07T08:58:31.382Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: c882c920-15d7-46ee-80d2-b3f957874ce3] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.291 ms [-]
identity_1     | [2019-01-20 20:35:54.264] cloudfoundry-identity-server - ???? [http-nio-8080-exec-404] .... DEBUG --- DisableIdTokenResponseTypeFilter: pre id_token disable:false pathinfo:null request_uri:/check_token response_type:null
periscope_1    | 2019-01-30 07:12:10,005 [getThreadPoolExecutorFactoryBean-25] execute:50 INFO  c.s.p.m.AbstractMonitor - [owner:periscope] [id:] [cb-stack-id:] Succesfully submitted com.sequenceiq.periscope.monitor.evaluator.CronTimeEvaluator for cluster 2.
cloudbreak_1   | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
uluwatu_1      | 2019-01-07T08:59:31.450Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: d1a3b046-58bb-4200-8481-17c591a10c5a] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.574 ms [-]
identity_1     | [2019-01-20 20:35:54.264] cloudfoundry-identity-server - ???? [http-nio-8080-exec-404] .... DEBUG --- DisableIdTokenResponseTypeFilter: post id_token disable:false pathinfo:null request_uri:/check_token response_type:null
cloudbreak_1   | 	at java.lang.Thread.run(Thread.java:748)
identity_1     | [2019-01-20 20:35:54.264] cloudfoundry-identity-server - ???? [http-nio-8080-exec-404] .... DEBUG --- SecurityFilterChainPostProcessor$HttpsEnforcementFilter: Filter chain 'checkTokenSecurity' processing request POST /check_token
cloudbreak_1   | 2019-01-30 12:28:24,544 [reactorDispatcher-73] fireCloudbreakEvent:58 INFO  c.s.c.s.e.DefaultCloudbreakEventService - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACKVIEW] [id:2] [name:hbaseha] [flow:cee721f4-1a72-407b-a1f9-984dd4aa99f5] [tracking:] Firing Cloudbreak event: entityId: 2, type: UPDATE_FAILED, message: Stack update failed. Reason: The autoscale group's desired instance count is not match with the instance count in the cloudbreak. Desired count: 2 <> cb instance count: 0. Upscale is not possible!
periscope_1    | 2019-01-30 07:12:10,012 [getThreadPoolExecutorFactoryBean-17] execute:84 INFO  c.s.p.m.e.CronTimeEvaluator - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [id:2] [cb-stack-id:2] Finished cronTimeEvaluator for cluster 2 in 7 ms
uluwatu_1      | 2019-01-07T08:59:31.474Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: ee5ca965-cc3a-43a3-acf0-4558b1b49c90] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.294 ms [-]
cloudbreak_1   | 2019-01-30 12:28:24,544 [reactorDispatcher-73] sendEvent:126 INFO  c.s.c.c.f.AbstractAction - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACKVIEW] [id:2] [name:hbaseha] [flow:cee721f4-1a72-407b-a1f9-984dd4aa99f5] [tracking:] Triggering event: UPSCALEFAILHANDLED, payload: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@75678d26
identity_1     | [2019-01-20 20:35:54.346] cloudfoundry-identity-server - ???? [http-nio-8080-exec-404] ....  INFO --- Audit: ClientAuthenticationSuccess ('Client authentication success'): principal=cloudbreak, origin=[remoteAddress=172.17.0.1, clientId=cloudbreak], identityZoneId=[uaa]
periscope_1    | 2019-01-30 07:12:15,257 [Timer-1] collectStackDetails:54 INFO  c.s.p.s.StackCollectorService - [owner:spring] [id:] [cb-stack-id:] Evaluate cluster management for stack: ad-tech-user (ID:6)
uluwatu_1      | 2019-01-07T09:00:01.374Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: b6a3569e-0cf6-4949-890e-b102d808726a] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.457 ms [-]
cloudbreak_1   | 2019-01-30 12:28:24,544 [reactorDispatcher-54] buildLogContextForReactorHandler:69 INFO  c.s.c.l.LogContextAspects - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACKVIEW] [id:2] [name:hbaseha] [flow:cee721f4-1a72-407b-a1f9-984dd4aa99f5] [tracking:] A Reactor event handler's 'accept' method has been intercepted: execution(CloudbreakEventHandler.accept(..)), MDC logger context is built.
periscope_1    | 2019-01-30 07:12:15,258 [Timer-1] collectStackDetails:58 INFO  c.s.p.s.StackCollectorService - [owner:spring] [id:] [cb-stack-id:] Succesfully submitted, the stack id: 6. Executor: java.util.concurrent.ThreadPoolExecutor@287acac2[Running, pool size = 50, active threads = 0, queued tasks = 1, completed tasks = 2692316]
uluwatu_1      | 2019-01-07T09:00:31.537Z INFO [owner: cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [email: ] [tracking: 5ca73cf2-8e69-4691-9ee4-66f498ab868b] ::ffff:172.17.0.1 - POST /notifications HTTP/1.1 200 0.442 ms [-]
identity_1     | [2019-01-20 20:36:13.147] cloudfoundry-identity-server - ???? [ZoneAwareMetadataManager.Refresh[idpMetadataManager]] .... DEBUG --- ZoneAwareIdpMetadataManager: Running SAML SP refresh[ZoneAwareMetadataManager.Refresh[idpMetadataManager]-1226146693] - ignoreTimestamp=false

Re: cloudbreak Unable to resize a cluster for a particular component

New Contributor

hey we are using cloudbreak 2,9,0 , and logs are below

on.CloudConnectorException: The autoscale group's desired instance count is not match with the instance count in the cloudbreak. Desired count: 2 <> cb instance count: 0. Upscale is not possible!
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.handler.UpscaleValidationHandler.accept(UpscaleValidationHandler.java:43)
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.handler.UpscaleValidationHandler.accept(UpscaleValidationHandler.java:20)
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.handler.UpscaleValidationHandler$$FastClassBySpringCGLIB$$c2eb296b.invoke(<generated>)
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.handler.UpscaleValidationHandler$$EnhancerBySpringCGLIB$$d5906469.accept(<generated>)
cloudbreak_1   | 2019-01-30 12:26:21,000 [reactorDispatcher-27] fireCloudbreakEvent:58 INFO  c.s.c.s.e.DefaultCloudbreakEventService - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACKVIEW] [id:2] [name:hbaseha] [flow:87cf4f4c-ad34-48db-aed1-17519da8f735] [tracking:] Firing Cloudbreak event: entityId: 2, type: UPDATE_FAILED, message: Stack update failed. Reason: The autoscale group's desired instance count is not match with the instance count in the cloudbreak. Desired count: 2 <> cb instance count: 0. Upscale is not possible!
cloudbreak_1   | 2019-01-30 12:26:21,001 [reactorDispatcher-27] sendEvent:126 INFO  c.s.c.c.f.AbstractAction - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACKVIEW] [id:2] [name:hbaseha] [flow:87cf4f4c-ad34-48db-aed1-17519da8f735] [tracking:] Triggering event: UPSCALEFAILHANDLED, payload: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@539f2b30
cloudbreak_1   | 2019-01-30 12:26:21,070 [reactorDispatcher-27] execute:87 INFO  c.s.c.c.f.AbstractAction - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACKVIEW] [id:2] [name:hbaseha] [flow:87cf4f4c-ad34-48db-aed1-17519da8f735] [tracking:] Stack: 2, flow state: UPSCALE_FAILED_STATE, phase: service, execution time 0 sec
cloudbreak_1   | 2019-01-30 12:26:21,071 [reactorDispatcher-27] stop:127 INFO  o.s.s.s.LifecycleObjectSupport - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACKVIEW] [id:2] [name:hbaseha] [flow:87cf4f4c-ad34-48db-aed1-17519da8f735] [tracking:] stopped EXTEND_HOST_METADATA_STATE EXTEND_HOST_METADATA_FINISHED_STATE INIT_STATE EXTEND_METADATA_FINISHED_STATE ADD_INSTANCES_STATE ADD_INSTANCES_FINISHED_STATE GATEWAY_TLS_SETUP_STATE EXTEND_METADATA_STATE BOOTSTRAP_NEW_NODES_STATE FINAL_STATE UPSCALE_FAILED_STATE UPSCALE_PREVALIDATION_STATE  /  / id=5cb13d82-da56-4bd1-b4c9-f0ed0045b6cc
cloudbreak_1   | 2019-01-30 12:28:21,195 [reactorDispatcher-63] start:107 INFO  o.s.s.s.LifecycleObjectSupport - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACK] [id:2] [name:hbaseha] [flow:499d11a1-7c42-4321-a206-9ca8a5ba5e97] [tracking:] started EXTEND_HOST_METADATA_STATE EXTEND_HOST_METADATA_FINISHED_STATE INIT_STATE EXTEND_METADATA_FINISHED_STATE ADD_INSTANCES_STATE ADD_INSTANCES_FINISHED_STATE GATEWAY_TLS_SETUP_STATE EXTEND_METADATA_STATE BOOTSTRAP_NEW_NODES_STATE FINAL_STATE UPSCALE_FAILED_STATE UPSCALE_PREVALIDATION_STATE  / INIT_STATE / id=5c4b2728-f0ff-4cc0-963b-fb2a17738ea2
cloudbreak_1   | 2019-01-30 12:28:21,402 [reactorDispatcher-63] getFirstValidPrivateId:219 INFO  c.s.c.c.f.s.u.StackUpscaleService - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACK] [id:2] [name:hbaseha] [flow:cee721f4-1a72-407b-a1f9-984dd4aa99f5] [tracking:] Get first valid PrivateId of instanceGroups
cloudbreak_1   | 2019-01-30 12:28:21,402 [reactorDispatcher-63] getFirstValidPrivateId:222 INFO  c.s.c.c.f.s.u.StackUpscaleService - [owner:cf5f7f0a-b3ae-45d6-98de-4de552bfc2d3] [type:STACK] [id:2] [name:hbaseha] [flow:cee721f4-1a72-407b-a1f9-984dd4aa99f5] [tracking:] Checking of instanceGroup: application