Cloudera Data Engineering (CDE) in CDP Public Cloud introduces a new bin-packing auto-scaling policy for more efficient resource allocation and cost reduction. Customers with bursty data pipelines with overlapping schedules can reap the cost benefits of faster scale down through the use of the new bin-packing policy.
Bin-packing policy is appropriate in situations where there's a mix of regular and bursty workloads deployed with some scheduling overlap. Usually, as nodes scale to meet the demand of a workload burst, there would be slow scale down because new jobs would be distributed across the new nodes preventing them from being freed up. Although this helps to reduce host spots, it leads to underutilization across the large number of nodes. With bin-packing as new jobs come online they are more efficiently allocated (ie - "bin-packed") to a smaller subset of nodes freeing the majority of the new nodes that were added.
This feature is available by default. Learn more here.
