About jchamp

jchamp · ‎09-06-2017

Thanks a lot for your feedback !! Yes we have currently set a long cooldown between scaling events and this seems ok. However we are investigating the following solution to remove this constraint, I'll be glad to discuss with you, or anyone interested on the potential following proposal. Separate HDFS on some nodes and "compute" on other nodes. For example we start a cluster with : - 3 hdfs nodes - 2 or 3 compute nodes And we use cloudbreak + ambari to create several alerts & scale policies : - 1 to scale up compute nodes ( based on % of allocated memory ) - 1 to scale down compute nodes ( based on the amount of remaining memory ) - 1 to scale up HDFS ( based on hdfs usage ) - 1 to scale down HDFS ( based on hdfs usage too ) Here I believe that we have a cluster which really scales regarding our requirements. Regards, Julien

jchamp · ‎08-30-2017

Hi I have a small cluster of 3 nodes with 40Go of memory I've successfully been able to UP scale a cluster using a slightly modified version of https://github.com/hortonworks/cloudbreak/blob/release-1.16/autoscale/src/main/resources/alerts/allocated_memory.ftl But I trigger an alert when the % of allocated memory > 95 % And this works perfectly : my cluster scales up when required. Now I want to do the ~reverse operation : scale down when 1 have one useless compute node : I'm going to try to create an alert when there's ~40Go of remaining memory free + 5% of 120 Go to prevent oscillations ( scale up / down / up down... ) Do you think this is a good way to solve my problem ? Thanks Julien

jchamp · ‎08-30-2017

Thanks @rkovacs I've been able to found a usefull solution thanks to you. I've used a slightly modified version of https://github.com/hortonworks/cloudbreak/blob/release-1.16/autoscale/src/main/resources/alerts/allocated_memory.ftl To trigger the UP scaling when the amount of allocated memory goes over 95% ( this triggers a critical alert ) I've just modified the aforementionned .ftl file to something more correct I think : we want an alert when the % of allocated memory goes over 95% and not when the % of remaining memory goes over 95% . I've changed : "value": "{0}/({0} + {1}) * 100" in : "value": "{1}/({0} + {1}) * 100" I'm going to open a new question about Scaling down now ! 🙂 i.e. : how to scale down 1 node is useless regarding to memory usage + a margin Thanks again !

jchamp · ‎08-25-2017

Hi, We have a cluster deployed using Cloudbreak and we want to use the Auto-Scaling feature of Cloudbreak. As read in the documentation it seems that we may be able to trigger a scale up using Cloudbreak "custom metrics: Pending YARN containers and Pending applications " ( c.f. : http://sequenceiq.com/cloudbreak-docs/latest/periscope/ ) However we haven't been able to find those metric in the webUi. In the form found in the webUi, in autoscaling SLA policies tab / Alerts / Metric desired states we can see many things () but nothing like "Pending YARN containers" nor "Pending Applications". Is there a good reason for that ? Is there a configuration problem ? ( we are able to send applications to YARN and everything is OK ) Do we need to do something to see this metric ? Install Something ? Configure something else ? Thx a lot for any idea / help ! Regards, Julien

Online	Offline
Last Visited	‎03-27-2018 07:22 AM

Date Registered	‎08-25-2017 03:22 PM
Last Visited	‎03-27-2018 07:22 AM
Total Messages Posted	4

Re: [CloudBreak] Scale DOWN when 1 node is useless...

[CloudBreak] Scale DOWN when 1 node is useless acc...

Re: [CloudBreak] Use Pending YARN containers OR Pe...

[CloudBreak] Use Pending YARN containers OR Pendin...