Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

unable to resize cluster

Highlighted

unable to resize cluster

Explorer

Unable to downsize the cluster. I tied to remove 45 worker nodes of 60 node cluster. Used the resize option to do it.Its been more than 2hrs and cluster is still in `update In Progress` status.

In ambari-server.log I see that ambari is trying to get metrics on deleted hosts

------------

21 Mar 2018 03:13:03,886 WARN [alert-event-bus-2] AlertReceivedListener:497 - Unable to process alert ambari_agent_disk_usage for cluster prod-dataeng-mr-2018-3-19 and host ip-*-*-*-*.us-west-2.compute.internal because the host is not a part of the cluster.

-----------------

Thanks in Asvance

4 REPLIES 4

Re: unable to resize cluster

Expert Contributor

Hi,

Are these worker nodes running Datanode? If they do Cloudbreak tries to decommission these nodes which most likely will never succeed as it needs to move data from 45 nodes to the remaining 15 nodes. Can you take a look at the CB logs to see what Cloudbreak is waiting for?

Highlighted

Re: unable to resize cluster

@prarthana basgod

Do you have any updates on this issue?

Highlighted

Re: unable to resize cluster

Explorer

logs did not have much info

```/cbreak_cloudbreak_1 | 2018-03-21 02:06:00,950 [reactorDispatcher-14] accept:52 INFO c.s.c.c.h.DownscaleStackHandler - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACK] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] Received event: Event{id=null, headers={FLOW_CHAIN_ID=cdce6305-ecb8-415a-845a-a11097f4cc4e, FLOW_ID=83720a79-6362-4773-b68f-e3539f052aab, MDC_CONTEXT_ID={owner=138893ef-307c-4584-96a2-512b2e25ec50, resourceId=31, resourceName=prod-dataeng-mr-2018-3-19, flowId=83720a79-6362-4773-b68f-e3539f052aab, resourceType=STACK}}, replyTo=null, key=DOWNSCALESTACKREQUEST, data=CloudStackRequest{, cloudStack=CloudStack{groups=[Group{name='client', type=CORE, instances=[CloudInstance{instanceId='i-04f395d8d77145523', template=InstanceTemplate{flavor='d2.xlarge', groupName='client', privateId=0, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@267b8064}], security=Security{rules=[]}, publicKey='null', loginUserName='cloudbreak', instanceAuthentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@5b6a16b0, skeleton=Optional.empty}, Group{name='master1', type=CORE, instances=[CloudInstance{instanceId='i-0a4c6cf679bfa2231', template=InstanceTemplate{flavor='d2.xlarge', groupName='master1', privateId=1, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@2763ca4}], security=Security{rules=[]}, publicKey='null', loginUserName='cloudbreak', instanceAuthentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@1977230f, skeleton=Optional.empty}, Group{name='master2', type=CORE, instances=[CloudInstance{instanceId='i-00aed7c5c1dced5d4', template=InstanceTemplate{flavor='d2.xlarge', groupName='master2', privateId=2, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@6375e8e5}], security=Security{rules=[]}, publicKey='null', loginUserName='cloudbreak', instanceAuthentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@68a1bdad, skeleton=Optional.empty}, Group{name='master3', type=GATEWAY, instances=[CloudInstance{instanceId='i-0b346b03b51888ad9', template=InstanceTemplate{flavor='d2.xlarge', groupName='master3', privateId=3, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@5b812f01}], security=Security{rules=[]}, publicKey='null', loginUserName='cloudbreak', instanceAuthentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@7bce1025, skeleton=Optional.empty}, Group{name='task', type=CORE, instances=[CloudInstance{instanceId='i-0c3680df4f6f49ed5', template=InstanceTemplate{flavor='d2.xlarge', groupName='task', privateId=4, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@37b0ed79}], security=Security{rules=[]}, publicKey='null', loginUserName='cloudbreak', instanceAuthentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@521972f7, skeleton=Optional.empty}, Group{name='worker', type=CORE, instances=[CloudInstance{instanceId='i-0311595a86edc9e42', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=20, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@b2b692}, CloudInstance{instanceId='i-03e88153bdadbe48b', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=55, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@bf6d199}, CloudInstance{instanceId='i-0366df10d51bffe23', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=58, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@3b9d3ebb}, CloudInstance{instanceId='i-0e2409f68a43b1e3c', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=33, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@220a7df5}, CloudInstance{instanceId='i-05567f7211971a69f', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=35, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@37195900}, CloudInstance{instanceId='i-0f5a7ed3138188890', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=52, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@a5e3c63}, CloudInstance{instanceId='i-08383553af3f416ae', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=37, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@58e4b375}, CloudInstance{instanceId='i-01705049c2cc52210', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=53, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@29709849}, CloudInstance{instanceId='i-087c442bd8dc51cab', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=28, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@545c053e}, CloudInstance{instanceId='i-05815cc555f44e8c0', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=36, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@2f1a96f2}, CloudInstance{instanceId='i-0ffbae91fbd67f37c', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=30, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@5298e77f}, CloudInstance{instanceId='i-0d630b26265630e59', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=45, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@7f078fb5}, CloudInstance{instanceId='i-0d98b652f0d7f185a', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=19, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@4bc6009e}, CloudInstance{instanceId='i-00665e7f605bc68e9', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=13, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@3010b212}, CloudInstance{instanceId='i-0f5d18a3e71e1f51f', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=9, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@25261870}, CloudInstance{instanceId='i-00b3c8db67eaf590d', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=59, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@661f4146}, CloudInstance{instanceId='i-067c096f0cb4f204a', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=44, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@1d267feb}, CloudInstance{instanceId='i-0f2275346bbdd5e86', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=22, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@2893ce7f}, CloudInstance{instanceId='i-0cfa95ebf3971cbb2', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=41, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@3d63a685}, CloudInstance{instanceId='i-0a9dae60b1ff2931a', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=25, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@21ead51e}, CloudInstance{instanceId='i-01405afc559df1237', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=12, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@f9d2fef}, CloudInstance{instanceId='i-023b250c84ce3a72a', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=7, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@394be855}, CloudInstance{instanceId='i-0f1f2c22ee7955f21', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=46, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@19bc2677}, CloudInstance{instanceId='i-0dbf819a4a289e102', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=31, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@6e89427b}, CloudInstance{instanceId='i-0407e3071f681930b', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=24, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@80dfa12}, CloudInstance{instanceId='i-032429422919778b5', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=32, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@6ee2b206}, CloudInstance{instanceId='i-0b6b0781d5a15beea', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=57, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@1695be33}, CloudInstance{instanceId='i-013d300310dc7d73c', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=47, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@322bbc00}, CloudInstance{instanceId='i-0be3893d5da0dcbe9', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=49, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@55910d3d}, CloudInstance{instanceId='i-0057b89ac770401ff', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=6, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@2e18f70b}, CloudInstance{instanceId='i-06a045692a232d05c', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=27, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@15c7a0dd}, CloudInstance{instanceId='i-09621c2ea83b35e73', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=42, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@7a63d5d9}, CloudInstance{instanceId='i-09d5bb7a11d659e33', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=51, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@2746c74d}, CloudInstance{instanceId='i-00610ab496d56ae54', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=10, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@4e8fdc01}, CloudInstance{instanceId='i-0333f01ab6ac95caf', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=56, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@66664c45}, CloudInstance{instanceId='i-04eda443c73ab0743', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=34, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@2cf804d}, CloudInstance{instanceId='i-09fb36a58db2d18f0', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=18, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@132243f6}, CloudInstance{instanceId='i-0387dffdf93046fd3', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=8, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@23fdfa47}, CloudInstance{instanceId='i-0bfa3f2dd7b5bc4d5', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=40, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@772d8e19}, CloudInstance{instanceId='i-03717e83ef9a9612a', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=50, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@91ea559}, CloudInstance{instanceId='i-04a6dfbd0a7bbe4e0', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=11, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@8227a7f}, CloudInstance{instanceId='i-0ed72656ec5906331', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=39, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@444186fe}, CloudInstance{instanceId='i-00ecf12e9c45dcf57', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=21, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@5f078529}, CloudInstance{instanceId='i-0f6a67fb3a725ab58', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=48, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@234fc7fd}, CloudInstance{instanceId='i-07cd2fc428ece4d31', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=38, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@3825a069}, CloudInstance{instanceId='i-0601845a9887918ee', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=14, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@709ba7e5}, CloudInstance{instanceId='i-0ea20b18a19feb9df', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=17, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@13565a12}, CloudInstance{instanceId='i-032cfe45db9767a68', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=43, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@4efb07aa}, CloudInstance{instanceId='i-07b8bab7bb5d94612', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=15, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@5bc8d0f4}, CloudInstance{instanceId='i-0f9c30e5c502efc0f', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=5, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@7641e0e7}, CloudInstance{instanceId='i-05b8c96128675793e', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=26, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@5766b2f7}, CloudInstance{instanceId='i-00794de2cdfd57495', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=54, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@41b629e6}, CloudInstance{instanceId='i-0e13ec65c154dbc67', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=16, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@5c641e54}, CloudInstance{instanceId='i-0be8e6baec12da1ee', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=23, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=DELETE_REQUESTED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@25db9be7}, CloudInstance{instanceId='i-01eae36fdcff71dd9', template=InstanceTemplate{flavor='d2.2xlarge', groupName='worker', privateId=29, volumes=[Volume{mount='/hadoopfs/fs1', type='gp2', size=50}], status=CREATED}, authentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@474864a1}], security=Security{rules=[]}, publicKey='null', loginUserName='cloudbreak', instanceAuthentication=com.sequenceiq.cloudbreak.cloud.model.InstanceAuthentication@37418cb7, skeleton=Optional.empty}], network=Network{subnet=com.sequenceiq.cloudbreak.cloud.model.Subnet@4b2bb2b5}, image=Image{imageName='ami-54bc022c', osType='redhat6', imageCatalogUrl='https://cloudbreak-imagecatalog.s3.amazonaws.com/v2-prod-cb-image-catalog.json', imageId='cab28152-f5e1-43e1-5107-9e7bbed33eef', imageCatalogName='cloudbreak-default', userdata={CORE=#!/bin/bash

/cbreak_cloudbreak_1 | 2018-03-21 02:06:00,951 [reactorDispatcher-14] authenticate:36 INFO c.s.c.c.a.AwsAuthenticator - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACK] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] Authenticating to aws ...
/cbreak_cloudbreak_1 | 2018-03-21 02:06:01,103 [reactorDispatcher-14] accept:70 ERROR c.s.c.c.h.DownscaleStackHandler - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACK] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] Failed to handle DownscaleStackRequest.
/cbreak_cloudbreak_1 | 2018-03-21 02:06:01,104 [reactorDispatcher-41] buildLogContextForReactorHandler:69 INFO c.s.c.l.LogContextAspects - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACK] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] A Reactor event handler's 'accept' method has been intercepted: execution(Flow2Handler.accept(..)), MDC logger context is built.
/cbreak_cloudbreak_1 | 2018-03-21 02:06:01,127 [reactorDispatcher-41] execute:87 INFO c.s.c.c.f.AbstractAction - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACKVIEW] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] Stack: 31, flow state: DOWNSCALE_STATE, phase: service, execution time 0 sec
/cbreak_cloudbreak_1 | 2018-03-21 02:06:01,127 [reactorDispatcher-41] handleStackDownscaleError:81 ERROR c.s.c.c.f.s.d.StackDownscaleService - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACKVIEW] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] Exception during the downscaling of stack
/cbreak_cloudbreak_1 | 2018-03-21 02:06:01,128 [reactorDispatcher-41] sendEvent:126 INFO c.s.c.c.f.AbstractAction - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACKVIEW] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] Triggering event: DOWNSCALEFAILHANDLED, payload: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@4b8becca
/cbreak_cloudbreak_1 | 2018-03-21 02:06:01,128 [reactorDispatcher-23] buildLogContextForReactorHandler:69 INFO c.s.c.l.LogContextAspects - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACKVIEW] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] A Reactor event handler's 'accept' method has been intercepted: execution(Flow2Handler.accept(..)), MDC logger context is built.
/cbreak_cloudbreak_1 | 2018-03-21 02:06:01,165 [reactorDispatcher-41] execute:87 INFO c.s.c.c.f.AbstractAction - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACKVIEW] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] Stack: 31, flow state: DOWNSCALE_FAILED_STATE, phase: service, execution time 0 sec
/cbreak_cloudbreak_1 | 2018-03-21 02:06:01,165 [reactorDispatcher-41] sendEvent:126 INFO c.s.c.c.f.AbstractAction - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACKVIEW] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] Triggering event: FLOWFINAL, payload: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@4b8becca

/cbreak_cloudbreak_1 | 2018-03-21 02:06:01,165 [reactorDispatcher-41] stop:127 INFO o.s.s.s.LifecycleObjectSupport - [owner:138893ef-307c-4584-96a2-512b2e25ec50] [type:STACKVIEW] [id:31] [name:prod-dataeng-mr-2018-3-19] [flow:83720a79-6362-4773-b68f-e3539f052aab] [tracking:] stopped org.springframework.statemachine.support.DefaultStateMachineExecutor@c9043fa```

I wasn't sure of future steps and had to terminate the cluster.

Highlighted

Re: unable to resize cluster

@prarthana basgod

I agree, this part of the logging is not verbose enough to find out the root cause. What is your CB version?

If you have a support subscription, feel free to open a ticket and we will happy to assist you in the further investigation, if not could you attach the whole cbreak.log, ambari-server.log and ambari-agent.log?

Don't have an account?
Coming from Hortonworks? Activate your account here