[CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

[#1]

Hi,

We have a cluster deployed using Cloudbreak and we want to use the Auto-Scaling feature of Cloudbreak.

As read in the documentation it seems that we may be able to trigger a scale up using Cloudbreak "custom metrics: Pending YARN containers and Pending applications" ( c.f. : http://sequenceiq.com/cloudbreak-docs/latest/periscope/ )

However we haven't been able to find those metric in the webUi.

In the form found in the webUi, in autoscaling SLA policies tab / Alerts / Metric desired states we can see many things () but nothing like "Pending YARN containers" nor "Pending Applications".

Is there a good reason for that ?

Is there a configuration problem ? ( we are able to send applications to YARN and everything is OK )

Do we need to do something to see this metric ? Install Something ? Configure something else ?

Thx a lot for any idea / help !

Regards,

Julien

[#2]

@Julien Champ

spun up a cluster using Cloudbreak 1.16.1. I do not see either of those options available. I will see if I can track this down and get clarification.

[#3]

Hi @Julien Champ,

This feature was remove near 1.14 and unfortunately the documentation was not updated as well.

[#4 · Accepted solution]

Hi @Julien Champ,

So Cloudbreak doesn't configures that metric by default, but you should use Ambari API to enable it. Simply grab your JSON template from here, and POST it to Ambari on clusters/[cluster-name]/alert_definitions endpoint.

[#5]

Thanks @rkovacs I've been able to found a usefull solution thanks to you.

I've used a slightly modified version of https://github.com/hortonworks/cloudbreak/blob/release-1.16/autoscale/src/main/resources/alerts/allo...

To trigger the UP scaling when the amount of allocated memory goes over 95% ( this triggers a critical alert )

I've just modified the aforementionned .ftl file to something more correct I think : we want an alert when the % of allocated memory goes over 95% and not when the % of remaining memory goes over 95% .

I've changed :

"value": "{0}/({0} + {1}) * 100"

in :

"value": "{1}/({0} + {1}) * 100"

I'm going to open a new question about Scaling down now ! 🙂

i.e. : how to scale down 1 node is useless regarding to memory usage + a margin

Thanks again !