Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Problem with Downscaling a node from cluster using Periscope.

Highlighted

Problem with Downscaling a node from cluster using Periscope.

New Contributor

Hi All,

I have been using Cloudbreak for deploying my cluster in AWS intensively. Recently, I was trying to use Periscope auto-scaling feature. This feature is very essential for our organisation as the cluster needs to autoscale based on certain yarn metrics. A more detailed description of my problem is as below,

I use a ten node cluster with replication factor 3. I have 7 slave nodes running in my cluster. When I scale up using the metrics provided by Periscope I found it very smoothly scaling up, elegantly.

The problem occurs when I try to downscale. The first time I downscale by one node, the decommissioning happens smoothly and the node gets removed. The next request to scale down by one node triggers the downscale operation successfully but, fails internally with the following error message "Text must not be null" with an IllegalArgumentException. I have attached the stacktrace in the file attached, a short description about the exception is below,

cloudbreak_1 | 2016-01-15 16:14:08,107 [reactorDispatcher-9] accept:38 INFO c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Sending notification with data: CloudbreakEvent{id=467, eventType='AVAILABLE', eventTimestamp=Fri Jan 15 16:14:08 UTC 2016, eventMessage='New node(s) could not be removed to the cluster. Reason java.lang.IllegalArgumentException: Text must not be null', owner='1b534ac9-8c83-4f18-84d2-515caca7f958', account='seq1234567', cloud='AWS', region='AP_SOUTHEAST_1', blueprintName='hdp-small-default', instanceGroup='null', blueprintId=4, stackId=12}

According to my judgement, the instanceGroup field is set to null internally eventhough, I mention to scale down a slave node. Can some one guide through on how to fix this issue? Your help will be much appreciated.

Thank you.

Best,

Ramprasad S

Big data Engineer - Affle

Singapore.

short-stacktrace.txt

5 REPLIES 5
Highlighted

Re: Problem with Downscaling a node from cluster using Periscope.

Mentor

@Ramprasad Sivaprakasam see your blueprint whether your slave node is identified by an instance group. Instance Groups.

Highlighted

Re: Problem with Downscaling a node from cluster using Periscope.

New Contributor

Hi @ArtemErvits ,

Thanks for the quick response. No, I did not add any instance group to my blueprint for my slave nodes. Could you help me with, where should I add the instance group in the blueprint?

Re: Problem with Downscaling a node from cluster using Periscope.

Mentor

I'm merely suggesting you look at the api and pass a parameter to identify a resource group like core, master and task, firstly you need to identify which resource group it belongs to of course.

Highlighted

Re: Problem with Downscaling a node from cluster using Periscope.

Mentor

@Ramprasad Sivaprakasam please view the README for Cloudbreak shell. I remember using this CLI for up/down scaling Cloudbreak. It has a specific mention about instance groups.

You have to configure the instancegroups before the provisioning. An instancegroup is defining a group of your nodes with a specified templates.

<code>instancegroup configure --instanceGroup slave_1 --nodecount 1 --templateId 51

Other available options:

  • --templateName "string": Name of the template
Highlighted

Re: Problem with Downscaling a node from cluster using Periscope.

Mentor

@Ramprasad Sivaprakasam are you still having issues with this? Can you accept best answer or provide your own solution?

Don't have an account?
Coming from Hortonworks? Activate your account here