Reply
New Contributor
Posts: 6
Registered: ‎08-04-2017

Support for new AWS partition placement groups

[ Edited ]

Amazon has recently introduced partition placement groups

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#placement-groups-partition

which finally solves the problem of multiple HDFS nodes being placed on the same underlying physical hardware.

 

Is it on the roadmap to add support for this to director ?

Cloudera Employee
Posts: 79
Registered: ‎02-18-2014

Re: Support for new AWS partition placement groups

Hi john1,

 

To my surprise, I think Altus Director can already work with partition placement groups. It is already possible to name a placement group for EC2 instances in an Altus Director instance template. From my interpretation of the AWS docs, if that placement group has a "partition" strategy, instances can be placed into it even without specifying a partition number (and today, Altus Director doesn't support picking a partition number).

 

I don't know how EC2 decides which partition in the placement group each instance should then be placed into. Maybe it's random or round robin, which might be good enough for spreading out HDFS datanodes if there are enough partitions. I encourage you to give it a shot and see what happens.

 

Explicit support isn't on the roadmap right now for Altus Director, but at first glance it doesn't seem too difficult to add it. The AWS plugin is open source, so you could also try adding it in yourself. Essentially, it would involve adding a new configuration property for EC2InstanceTemplate for the partition number and then including its value in RunInstanceRequest objects.

 

Good luck! If you do some experiments, I'd be interested to know how they turn out.

 

Bill