question Re: [CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling in Support Questions

[CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

jchamp — Fri, 25 Aug 2017 22:45:54 GMT

Hi,

We have a cluster deployed using Cloudbreak and we want to use the Auto-Scaling feature of Cloudbreak.

As read in the documentation it seems that we may be able to trigger a scale up using Cloudbreak "custom metrics: Pending YARN containers and Pending applications" ( c.f. : http://sequenceiq.com/cloudbreak-docs/latest/periscope/ )

However we haven't been able to find those metric in the webUi.

In the form found in the webUi, in autoscaling SLA policies tab / Alerts / Metric desired states we can see many things () but nothing like "Pending YARN containers" nor "Pending Applications".

Is there a good reason for that ?

Is there a configuration problem ? ( we are able to send applications to YARN and everything is OK )

Do we need to do something to see this metric ? Install Something ? Configure something else ?

Thx a lot for any idea / help !

Regards,

Julien

Re: [CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

myoung — Fri, 25 Aug 2017 23:09:40 GMT

@Julien Champ

spun up a cluster using Cloudbreak 1.16.1. I do not see either of those options available. I will see if I can track this down and get clarification.

Re: [CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

rkovacs — Mon, 28 Aug 2017 14:06:46 GMT

Hi @Julien Champ,

This feature was remove near 1.14 and unfortunately the documentation was not updated as well.

Re: [CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

rkovacs — Mon, 28 Aug 2017 19:30:55 GMT

Hi @Julien Champ,

So Cloudbreak doesn't configures that metric by default, but you should use Ambari API to enable it. Simply grab your JSON template from here, and POST it to Ambari on clusters/[cluster-name]/alert_definitions endpoint.

Re: [CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

jchamp — Wed, 30 Aug 2017 15:38:51 GMT

Thanks @rkovacs I've been able to found a usefull solution thanks to you.

I've used a slightly modified version of https://github.com/hortonworks/cloudbreak/blob/release-1.16/autoscale/src/main/resources/alerts/allocated_memory.ftl

To trigger the UP scaling when the amount of allocated memory goes over 95% ( this triggers a critical alert )

I've just modified the aforementionned .ftl file to something more correct I think : we want an alert when the % of allocated memory goes over 95% and not when the % of remaining memory goes over 95% .

I've changed :

"value": "{0}/({0} + {1}) * 100"

in :

"value": "{1}/({0} + {1}) * 100"

I'm going to open a new question about Scaling down now ! 🙂

i.e. : how to scale down 1 node is useless regarding to memory usage + a margin

Thanks again !