Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

Solved Go to solution

[CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

New Contributor

Hi,

We have a cluster deployed using Cloudbreak and we want to use the Auto-Scaling feature of Cloudbreak.

As read in the documentation it seems that we may be able to trigger a scale up using Cloudbreak "custom metrics: Pending YARN containers and Pending applications" ( c.f. : http://sequenceiq.com/cloudbreak-docs/latest/periscope/ )

However we haven't been able to find those metric in the webUi.

In the form found in the webUi, in autoscaling SLA policies tab / Alerts / Metric desired states we can see many things () but nothing like "Pending YARN containers" nor "Pending Applications".

Is there a good reason for that ?

Is there a configuration problem ? ( we are able to send applications to YARN and everything is OK )

Do we need to do something to see this metric ? Install Something ? Configure something else ?

Thx a lot for any idea / help !

Regards,

Julien

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: [CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

Expert Contributor

Hi @Julien Champ,

So Cloudbreak doesn't configures that metric by default, but you should use Ambari API to enable it. Simply grab your JSON template from here, and POST it to Ambari on clusters/[cluster-name]/alert_definitions endpoint.

View solution in original post

4 REPLIES 4
Highlighted

Re: [CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

@Julien Champ

spun up a cluster using Cloudbreak 1.16.1. I do not see either of those options available. I will see if I can track this down and get clarification.

Highlighted

Re: [CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

Expert Contributor

Hi @Julien Champ,

This feature was remove near 1.14 and unfortunately the documentation was not updated as well.

Highlighted

Re: [CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

Expert Contributor

Hi @Julien Champ,

So Cloudbreak doesn't configures that metric by default, but you should use Ambari API to enable it. Simply grab your JSON template from here, and POST it to Ambari on clusters/[cluster-name]/alert_definitions endpoint.

View solution in original post

Highlighted

Re: [CloudBreak] Use Pending YARN containers OR Pending application to trigger Cluster Upscaling

New Contributor

Thanks @rkovacs I've been able to found a usefull solution thanks to you.

I've used a slightly modified version of https://github.com/hortonworks/cloudbreak/blob/release-1.16/autoscale/src/main/resources/alerts/allo...

To trigger the UP scaling when the amount of allocated memory goes over 95% ( this triggers a critical alert )

I've just modified the aforementionned .ftl file to something more correct I think : we want an alert when the % of allocated memory goes over 95% and not when the % of remaining memory goes over 95% .

I've changed :

"value": "{0}/({0} + {1}) * 100"

in :

"value": "{1}/({0} + {1}) * 100"

I'm going to open a new question about Scaling down now ! :)

i.e. : how to scale down 1 node is useless regarding to memory usage + a margin

Thanks again !

Don't have an account?
Coming from Hortonworks? Activate your account here